Web Crawler

Node.js and databases Teamwork assignment- https://in.thecamp.me/resource/-Project-Assignment/4Z2SvRlg/JS-Nov-17-Module-3/6qZhL%2BTF

App Description

• Node.js CLI application
• scrapes http://www.technopolis.bg & https://smartphone.bg
• no APIs, parse each url with jsdom and jquery
• parses all of the phones’ urls again with jsdom and jquery
• ignores useless data, stores the important data
• the application is based on asynchronous operations using Promises and Async/Awaits.

Available commands:

npm run

• statistics (filter the data, prints the result)

  • filter: ram :method:value
  • search:details_column:value

  • order-by-brand:value (asc

    desc)

• show-data (prints everything from the database)

• reset (delete the crawled data)

Dependancies

• jQuery
• jsdom
• mysql
• mysql2
• sequelize
• sequelize-auto-migrations