You should exercise whenever you have time. If you are free the entire day the best time would be early in the day after you've had something to eat.
After you wake up and eat breakfast, you're energized and you'll probably get the best workout in. If you decide to go when you get off work in the afternoon, you might be too tired to go and if you make it to the gym, you might not work as hard as you'd like.
Nighttime workouts have the same problem. After eating a large meal such as a typical dinner, blood rushes to your stomach to aid in digestion. This can make you feel tired and will cause you to have a bad workout.
Unfortunately, most of us don't have all day, so exercising even at the worst time is better than not exercising at all.