V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
V2EX 提问指南
U87
V2EX  ›  问与答

明明有 key 啊,为啥报错 KeyError

  •  
  •   U87 · 2018-05-01 15:09:33 +08:00 · 2342 次点击
    这是一个创建于 2400 天前的主题,其中的信息可能已经有所发展或是发生改变。

    class QingHaiSpider(scrapy.Spider): name = "qinghai" headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/537.36', 'Accept': 'application/json, text/javascript, /; q=0.01'} results = {"001002":"01", "001001":"02", "001005":"03"} def start_requests(self): url = 'http://111.44.251.34/inteligentsearch/rest/inteligentSearch/getFullTextData' for k in self.results: formdata1 = {"token":"","pn":0,"rn":10,"sdt":"","edt":"","wd":"","inc_wd":"","exc_wd":"","fields":"title","cnum":"001;002;003;004;005;006;007;008;009;010","sort":"{"showdate":"0"}","ssort":"title","cl":200,"terminal":"","condition":[{"fieldName":"categorynum","isLike":True,"likeType":2,"equal":k}],"time":None,"highlights":"title","statistics":None,"unionCondition":None,"accuracy":"100","noParticiple":"0","searchRange":None,"isBusiness":1} formdata = json.dumps(formdata1) request = scrapy.Request(url=url, method='POST', body=formdata ,headers=self.headers, callback=self.parse, dont_filter=False, meta={'key':k})

            yield request
    
    
    def parse(self, response):
        w = response.meta['key']
        max_counts = json.loads(response.text)["result"]["totalcount"]
        datas = json.loads(response.text)["result"]["records"]
        for data in datas:
            item = QhItem()
            item["title"] = data["title"]
            item["date"] = data["showdate"]
            item["detail_url"] = "http://111.44.251.34" + data["linkurl"]
            item["area_code"] = "QINGHAI"
            item["publish_id"] = "181818"
            item["thing_id"] = "42"
            if "001002" in item["detail_url"]:
                item["content_type"] = "01"
    
            elif "001001" in item["detail_url"]:
                item["content_type"] = "02"
            else:
                item["content_type"] = "03"
    
            yield item
    
    
        for i in range(10, max_counts + 10, 10):
    
            formdata2 = {"token":"","pn":i,"rn":10,"sdt":"","edt":"","wd":"","inc_wd":"","exc_wd":"","fields":"title","cnum":"001;002;003;004;005;006;007;008;009;010","sort":"{\"showdate\":\"0\"}","ssort":"title","cl":200,"terminal":"","condition":[{"fieldName":"categorynum","isLike":True,"likeType":2,"equal":w}],"time":None,"highlights":"title","statistics":None,"unionCondition":None,"accuracy":"100","noParticiple":"0","searchRange":None,"isBusiness":1}
            formdata = json.dumps(formdata2)
            url = 'http://111.44.251.34/inteligentsearch/rest/inteligentSearch/getFullTextData'
            yield scrapy.Request(url=url, method='POST', body=formdata ,headers=self.headers, callback=self.parse, dont_filter=False)
    
    目前尚无回复
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   1598 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 24ms · UTC 16:57 · PVG 00:57 · LAX 08:57 · JFK 11:57
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.