Subnetting used to be known as classes. You would have A, B en C classes. Firms used to be given a class, let's say "IBM" would have gotten a A-class subnet, which would give a default subnetmask like 255.0.0.0, which of course nowadays is utterly insane cause of the insane great network and hosts.
A-class was a default subnet mask of 255.0.0.0, B-class 255.255.0.0, C-class 255.255.255.0. They used to decide which class you would get by how big your company was/is.

VLSM is the possibility of dividing for.ex a C-class subnet (255.255.255.0) into different variable subnets.

Example:

You can divide this subnet in for.ex. an 255.255.255.128 subnet (128 hosts, actually 126 usable) and 2 255.255.255.192 subnets each consisting out of (64 - 2) = 62 usable hosts. There are of course lots of other combinations. This one is just to show you the meaning of vlsm.

192.168.0.0/255.255.255.128 =128 - 2 (first and last) = 126 usable ip's
192.168.0.128/255.255.255.192 = 64 - 2 (first and last) = 62 usable ip's
192.168.0.192/255.255.255.192 = 64 - 2 (first and last) = 62 usable ip's

This would divide that c-class in 3 different subnets giving a better usage, and making the networks fit your needs, since also the demand for ip's increased alot the last decade and if the old way of subnetting would be holded we would have run out of ip's nowadays. Therefor the vlsm and ofcourse the coming of ipv6 which will replace the ipv4. The vlsm its purpose actually was/is a better administration of public ip adresses cause int the ipv4 the free ranges are getting scarse.

Cacosapo, RIP_v2 should be able to handle vlsm, as also others like ospf and eigrp.

Here is a good tool to help your subnetting : Click here