For most you can right click on the icons and chose Exit, or Close. However, make sure you are not closing down things you actually -need- (like a firewall or antivirus).

Depending upon your version of Windows, you can also:

Start > Run.. > msconfig

And use the tab on the far topright called Startup to stop certain things from automagically loading on your computer. If you do not know what some of the items listed there do, either google them or post here for help and we can google for you.

Many third party programs exist, but for now msconfig will do what you need without hassel