我正在按照外部文件中的示例从外部文件运行爬行蜘蛛。我想抓取Core API提供的统计信息，并在抓取完成后将其存储到mysql表中。
from twisted.internet import reactor
from scrapy.crawler import Crawler
from scrapy import log, signals
from test.spiders.myspider import *
from scrapy.utils.project import get_project_settings
from test.pipelines import MySQLStorePipeline
import datetime
spider = MySpider()
def run_spider(spider):
settings = get_project_settings()
crawler = Crawler(settings)
crawler.signals.connect(reactor.stop, signal=signals.spider_closed)
crawler.configure()
crawler.crawl(spider)
crawler.start()
log.start()
reactor.run()
mysql_insert = MySQLStorePipeline()
mysql_insert.cursor.execute(
'insert into crawler_stats(sites_id, start_time,end_time,page_scraped,finish_reason)
values(%s,%s,%s, %s,%s)',
(1,datetime.datetime.now(),datetime.datetime.now(),100,'test'))
mysql_insert.conn.commit()
run_spider(spider)
如何在上面的代码中获取start_time，end_time，pages_scraped，finish_reason等统计信息的值？
从crawler.stats收藏家那里获取它们：
stats = crawler.stats.get_stats()
示例代码（在spider_closed信号处理程序中收集统计信息）：
def callback(spider, reason):
stats = spider.crawler.stats.get_stats() # stats is a dictionary
# write stats to the database here
reactor.stop()
def run_spider(spider):
settings = get_project_settings()
crawler = Crawler(settings)
crawler.signals.connect(callback, signal=signals.spider_closed)
crawler.configure()
crawler.crawl(spider)
crawler.start()
log.start()
reactor.run()
run_spider(spider)
本文收集自互联网，转载请注明来源。
如有侵权，请联系[email protected] 删除。
我来说两句