資源簡介
由于項目需要，特研究了一段時間關于java爬蟲的相關技術，發現一個比較好用的爬蟲框架--WebMagic，只需少量代碼即可實現一個爬蟲，本項目就是基于它的一個簡單實現，導入項目即可運行，項目只有兩個類，一個用于抓取，一個用于處理抓取到的數據，存入數據庫或導出到excel等(只打印到控制臺，后續自己發揮)，簡單吧，代碼真的很少
代碼片段和文件信息
package?pipeline;
import?java.util.Map;
import?org.slf4j.Logger;
import?org.slf4j.LoggerFactory;
import?us.codecraft.webmagic.ResultItems;
import?us.codecraft.webmagic.Task;
import?us.codecraft.webmagic.pipeline.Pipeline;
public?class?HdPipeline?implements?Pipeline?{
??private?Logger?logger?=?LoggerFactory.getLogger（HdPipeline.class）;
??@Override
??public?void?process（ResultItems?resultItems?Task?task）?{
??//處理抓取到的數據或存入數據庫
??Map?resultMap?=?（Map）?resultItems.get（“resultMap“）;
??System.out.println（resultMap）;
???
??}
}
?屬性????????????大小?????日期????時間???名稱
-----------?---------??----------?-----??----
?????目錄???????????0??2016-12-14?10:22??spider\
?????文件????????2829??2016-12-14?16:13??spider\.classpath
?????文件?????????382??2016-12-14?10:21??spider\.project
?????目錄???????????0??2016-12-14?10:24??spider\.settings\
?????文件?????????103??2016-12-14?12:08??spider\.settings\org.eclipse.core.resources.prefs
?????文件?????????598??2016-12-14?10:21??spider\.settings\org.eclipse.jdt.core.prefs
?????目錄???????????0??2016-12-14?16:13??spider\bin\
?????目錄???????????0??2016-12-14?16:13??spider\bin\pipeline\
?????文件????????1036??2016-12-23?11:28??spider\bin\pipeline\HdPipeline.class
?????目錄???????????0??2016-12-23?11:27??spider\bin\processor\
?????文件????????8282??2016-12-23?11:33??spider\bin\processor\HdProcessor.class
?????目錄???????????0??2016-12-14?16:12??spider\lib\
?????文件??????576338??2016-12-14?10:22??spider\lib\assertj-core-1.5.0.jar
?????文件?????2902942??2016-09-29?10:03??spider\lib\bcprov-jdk15on-1.52.jar
?????文件??????188671??2016-12-14?16:12??spider\lib\commons-beanutils-1.7.0.jar
?????文件??????232771??2016-12-14?10:22??spider\lib\commons-codec-1.6.jar
?????文件??????259600??2016-09-29?10:10??spider\lib\commons-codec-1.7.jar
?????文件??????559366??2016-12-14?16:12??spider\lib\commons-collections-3.1.jar
?????文件??????575389??2016-12-14?10:22??spider\lib\commons-collections-3.2.1.jar
?????文件???????87776??2016-12-14?10:22??spider\lib\commons-io-1.3.2.jar
?????文件??????159509??2016-09-01?10:57??spider\lib\commons-io-2.0.1.jar
?????文件??????279193??2016-12-14?16:12??spider\lib\commons-lang-2.5.jar
?????文件??????284220??2016-12-14?10:22??spider\lib\commons-lang-2.6.jar
?????文件??????315805??2016-12-14?10:22??spider\lib\commons-lang3-3.1.jar
?????文件???????62050??2016-12-14?10:22??spider\lib\commons-logging-1.1.3.jar
?????文件???????61829??2016-08-18?14:33??spider\lib\commons-logging-1.2.jar
?????文件???????45405??2016-12-14?16:12??spider\lib\commons-logging.jar
?????文件???????96221??2016-08-10?16:37??spider\lib\commons-pool-1.5.4.jar
?????文件??????100193??2016-12-14?10:22??spider\lib\commons-pool-1.5.5.jar
?????文件???????77826??2016-12-14?16:12??spider\lib\ezmorph-1.0.3.jar
?????文件??????356654??2016-12-14?10:22??spider\lib\fastjson-1.1.37.jar
............此處省略26個文件信息
評論
共有 條評論
川公網安備 51152502000135號