How do families influence the ability of children? Cognitive skills have been shown to be a strong predictor of educational attainment and future labor market success; as a result, understanding the determinants of cognitive skills can lead to a better understanding of children’s long run outcomes. This paper uses a large dataset on the male population of Norway and focuses on one family characteristic: the effect of family size on IQ. Because of the endogeneity of family size, we instrument for family size using twin births and sex composition. IV estimates using sex composition as an instrument show no negative effect of family size; however, IV estimates using twins imply that family size has a negative effect on IQ. Our results suggest that effect of family size depends on the type of family size intervention. We conclude that there are no important negative effects of expected increases in family size on IQ but that unexpected shocks to family size resulting from twin births have negative effects on the IQ of existing children.