Previous work suggests that self–other agreement in personality judgment is lower for evaluative than for neutral items. We tested whether this pattern generalizes to judgments collected with Who Knows, a mobile app in which users watch brief video clips of strangers and try to predict how targets answered highly concrete personality statements. The dataset comprised >340,000 predictions from >4,000 judges across 296 items and 75 targets. Contrary to prior findings, item evaluativeness did not reduce prediction accuracy with respect to targets’ self-reports. Instead, evaluativeness was positively associated with prediction accuracy for Likert-type items while no effect was observed for binary items. Other coded item features (e.g., base rate and psychological modality) showed no consistent associations with prediction accuracy across formats. We propose two boundary conditions that may help reconcile these results with earlier work: (a) the present item pool is unusually concrete, which alters the distribution and correlates of desirability/evaluativeness, and (b) judges were instructed to predict targets’ self-reports rather than provide their own impressions. Under this instruction, judges may partly rely on between-target differences in self-protective responding when forming predictions. These findings caution against transferring item-feature effects from adjective-based impression paradigms to concrete items and self-report prediction tasks.