Data Scraping

If you are looking to contribute to the CAN data scraping efforts, you have come to the right place!

In this section of the documentation you will learn how to get your local environment ready to start contributing and learn about our tooling and libraries

Contents:

• Developer Installation
  • Using Conda
• Structure
  • Organization
  • Class Hierarchy
  • Scraper Lifecycle
• Writing Scrapers
  • Finding the right subclass
  • Filling in class level attributes
  • Writing the fetch method
  • Writing the normalize method
  • Running the scraper locally
  • Running the tests for the scraper
• Deleting Data
  • SQL in Production
  • Timeseries scrapers
• FAQ
  • Location_id sql error
  • variable_id sql error
  • demographic_id sql error