Public health research has shown that neighborhood conditions are associated with health behaviors and outcomes. Systematic neighborhood audits have helped researchers measure neighborhood conditions that they deem theoretically relevant but not available in existing administrative data. Systematic audits, however, are expensive to conduct and rarely comparable across geographic regions. We describe the development of an online application, the Computer Assisted Neighborhood Visual Assessment System (CANVAS), that uses Google Street View to conduct virtual audits of neighborhood environments. We use this system to assess the inter-rater reliability of 187 items related to walkability and physical disorder on a national sample of 150 street segments in the United States. We find that many items are reliably measured across auditors using CANVAS and that agreement between auditors appears to be uncorrelated with neighborhood demographic characteristics. Based on our results we conclude that Google Street View and CANVAS offer opportunities to develop greater comparability across neighborhood audit studies.