91av视频/亚洲h视频/操亚洲美女/外国一级黄色毛片 - 国产三级三级三级三级

  • 大小: 1.59M
    文件類型: .zip
    金幣: 1
    下載: 0 次
    發(fā)布日期: 2021-01-30
  • 語言: Python
  • 標簽: 爬蟲??實例??

資源簡介

此項目的功能是爬取知乎用戶信息以及人際拓撲關系，爬蟲框架使用scrapy，數(shù)據(jù)存儲使用mongo，下載這些數(shù)據(jù)感覺也沒什么用，就當為大家學習scrapy提供一個例子吧。

## 使用方法

### 本地運行

爬蟲程序依賴mongo和rabbitmq，因此這兩個服務必須正常運行和配置。為了加快下載效率，圖片下載是異步任務，因此在啟動爬蟲進程執(zhí)行需要啟動異步worker，啟動方式是進入zhihu_spider/zhihu目錄后執(zhí)行下面命令:


代碼片段和文件信息 

#?-*-?coding=utf8?-*-
from?scrapy?import?cmdline

cmdline.execute（“scrapy?crawl?zhihu“.split（））

?屬性????????????大小?????日期????時間???名稱
-----------?---------??----------?-----??----
?????目錄???????????0??2016-03-20?14:42??zhihu_spider-master\
?????文件??????????30??2016-03-20?14:42??zhihu_spider-master\.gitignore
?????文件????????3056??2016-03-20?14:42??zhihu_spider-master\README.md
?????目錄???????????0??2016-03-20?14:42??zhihu_spider-master\doc\
?????文件??????753450??2016-03-20?14:42??zhihu_spider-master\doc\image.png
?????文件??????532877??2016-03-20?14:42??zhihu_spider-master\doc\people.png
?????文件??????413151??2016-03-20?14:42??zhihu_spider-master\doc\relation.png
?????文件??????126198??2016-03-20?14:42??zhihu_spider-master\doc\主頁.png
?????文件??????111628??2016-03-20?14:42??zhihu_spider-master\doc\代碼.png
?????文件????????3386??2016-03-20?14:42??zhihu_spider-master\doc\流程圖.graffle
?????文件???????97151??2016-03-20?14:42??zhihu_spider-master\doc\流程圖.png
?????文件?????????232??2016-03-20?14:42??zhihu_spider-master\docker-compose.yml
?????文件??????????60??2016-03-20?14:42??zhihu_spider-master\requirements.txt
?????目錄???????????0??2016-03-20?14:42??zhihu_spider-master\zhihu\
?????文件??????????96??2016-03-20?14:42??zhihu_spider-master\zhihu\main.py
?????文件?????????254??2016-03-20?14:42??zhihu_spider-master\zhihu\scrapy.cfg
?????目錄???????????0??2016-03-20?14:42??zhihu_spider-master\zhihu\zhihu\
?????文件???????????0??2016-03-20?14:42??zhihu_spider-master\zhihu\zhihu\__init__.py
?????文件?????????686??2016-03-20?14:42??zhihu_spider-master\zhihu\zhihu\constants.py
?????文件????????1092??2016-03-20?14:42??zhihu_spider-master\zhihu\zhihu\items.py
?????文件????????2536??2016-03-20?14:42??zhihu_spider-master\zhihu\zhihu\pipelines.py
?????文件????????3654??2016-03-20?14:42??zhihu_spider-master\zhihu\zhihu\settings.py
?????目錄???????????0??2016-03-20?14:42??zhihu_spider-master\zhihu\zhihu\spiders\
?????文件?????????161??2016-03-20?14:42??zhihu_spider-master\zhihu\zhihu\spiders\__init__.py
?????文件????????7657??2016-03-20?14:42??zhihu_spider-master\zhihu\zhihu\spiders\profile.py
?????目錄???????????0??2016-03-20?14:42??zhihu_spider-master\zhihu\zhihu\tools\
?????文件??????????44??2016-03-20?14:42??zhihu_spider-master\zhihu\zhihu\tools\__init__.py
?????文件?????????696??2016-03-20?14:42??zhihu_spider-master\zhihu\zhihu\tools\async.py

評論

共有 條評論