The ability trees are things like Battle Rage (all melee combat abilities\feats), Vitalism (all healing\buffing), Occult (opposite, harm\de-buff), sorcery (arcane power), shadowplay (all stealth, inviso, etc), archery (all ranged combat abilities\feats), etc etc.
So depending on how you combine the 3 trees and how many points you put into each tree, you can create nearly infinite custom classes. Some abilities still have pre-reqs, so you need to invest more points into some trees and you can't just cherry pick.
Radiance was a step closer to this idea, but it is still stuck in 'class abilities'. Imagine a system similar to Radiance, but with the depth of complexity in PF and an open ability system. It would be like having unlimited access to 3 full classes and not suffering the penalties of multiclassing. I know it sounds OP, but everyone still has essentially 20 levels, for example, to play with. So you could still get access to 8th level spells, but maybe only cast 1 per day instead of 3 and in exchange get 2 advanced Rogue talents, instead of 5.
some examples:
melee + defense + vitalism = paladin
melee + defense + occult = ill rigger
ranged + vitalism + stealth = harrier\ranger
shadow + melee + occult = sith warrior
occult + stealth + vitalism = vampiric necromancer
sorcery + melee + defense = battlemage
shadow + melee + ranged = assassin
vitalism + sorcery + occult = mystic theurge