A lot of people have anxiety about driving so don't think you are alone. For some people it is something specific such as driving on the motorway, in busy traffic, or unfamiliar roads. For others it's simply the thought of being in the driving seat and driving anywhere that starts those feelings of panic and anxiety.
If you're still learning to drive then finding a good driving instructor who understands anxious drivers will help to build your confidence. You will learn faster with someone who can feel comofortable with. There are driving instructors who specialise in teaching anxious drivers.
Get some practice in different conditions when you have some spare time. I learned to drive in the winter so nearly every lesson was in the dark and there were quite a few different weather conditions too. But, if you've learned to drive in the summer then bad weather and headlights could be something that you've never really had to deal with during your lessons. Getting some experience of them when you can take things at your own pace will help to build your confidence.
If you don't have much experience of driving in busy towns with multiple lanes then go out on a Sunday or late in the evening when there is less traffic so you can get used to changing lanes and all the distractions of traffic lights and signage. Then, if you go there again at a busier time it will be familiar and you will have less new information to take in.
When you are building up your confidence always set yourself tasks that you feel you can achieve, that way you will get used to driving and feeling able to do it. Even if to start off with you just go round the block once, it will leave you with a feeling of positive achievement that you wll associate with driving.
Remember all your positive experiences of driving because they will increase your confidence. The more familiar something becomes, the more comfortable you feel with it because it becomes more of an instinctive reflex.
Don't be intimidated by other drivers. You are in control of your car and know what you consider to be safe. Even if the person behind you is beeping their horn, don't feel you have to move or go faster if you don't think it is safe too. If someone is driving very close behind you and you feel uncomfortable then find a suitable place to pull in or pull over so they can pass you. Some people are just thoughtless and impatient, inside and outside of a car.
Always leave plenty of time for your journey so you don't feel anxious before you start. Leave about 50% extra time, so if your journey should take you 20 minutes, allow 30 so you know you still have time if you take a wrong turning or want to pull over for a while.
You can see someone for an individual session that is tailored to your own anxiety or you can find CDs and MP3 downloads that are more general but can still be helpful. There are also books and articles that have exercises which you can use yourself. In my experience most people find it easier to see someone for help because they get a different point of view which can allow your mind accept your confidence more easily.