There is also a nice dinosaur museum in the area, I'd especially check it out with kids.
This was the castle I visited just prior to Puilarens -- a much different style.
Château vicomtal Saint-Pierre de Fenouillet
This was the next castle I went to (not nearly impressive, but still nice).
Gorges de la Pierre-Lys
You'll drive through some beautiful scenery in this gorge on the way to Puilarens