Higher spin fields in four dimensions, and more generally conformal fields in arbitrary dimensions, can be described by spinning particle models with a gauged SO(N) extended supergravity on the worldline. We consider here the one-loop quantization of these models by studying the corresponding partition function on the one-dimensional torus. After gauge fixing the supergravity multiplet, the partition function reduces to an integral over the corresponding moduli space which is computed using orthogonal polynomial techniques. We obtain a compact formula which gives the number of physical degrees of freedom for all N in all dimensions. As an aside we compute the physical degrees of freedom of the SO(4) = SU(2)xSU(2) model with only a SU(2) factor gauged, which has attracted some interest in the literature.