推荐学习书目
› Learn Python the Hard Way
Python Sites
› PyPI - Python Package Index
› http://diveintopython.org/toc/index.html
› Pocoo
值得关注的项目
› PyPy
› Celery
› Jinja2
› Read the Docs
› gevent
› pyenv
› virtualenv
› Stackless Python
› Beautiful Soup
› 结巴中文分词
› Green Unicorn
› Sentry
› Shovel
› Pyflakes
› pytest
Python 编程
› pep8 Checker
Styles
› PEP 8
› Google Python Style Guide
› Code Style from The Hitchhiker's Guide
[求助] 为何微博网页代码有 1500 多行，而我只能提取 10%左右
15874103329 · 2018-12-23 11:49:25 +08:00 · 3086 次点击这是一个创建于 2167 天前的主题，其中的信息可能已经有所发展或是发生改变。
为何微博网页代码有 1500 多行（ response 内看到的），而我只能提取 10%左右，已经试过很多次了，代码如下：
import requests
from pyquery import PyQuery as pq
from urllib.parse import urlencode
import re
def dizhi():
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3573.0 Safari/537.36'}
data = {
'q': '微信群',
'typeall': '1',
'suball': '1',
'timescope': 'custom:2018 - 12 - 20 - 0: 2018 - 12 - 22 - 0',
'Refer': 'g'
}
url = 'https://s.weibo.com/weibo/%25E5%25AE%259D%25E5%25A6%2588%25E7%25BE%25A4?' + urlencode(data)
wangzhi = requests.get(url,headers = headers)
return wangzhi.text
def jiexi(html):
doc = pq(html)
item = doc('.m4 li')
for i in item.items():
print(i('img').attr('src'))
def main():
html = dizhi()
print(html)
jiexi(html)
if __name__ == '__main__':
main()
import requests
from pyquery import PyQuery as pq
from urllib.parse import urlencode
import re
def dizhi():
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3573.0 Safari/537.36'}
data = {
'q': '微信群',
'typeall': '1',
'suball': '1',
'timescope': 'custom:2018 - 12 - 20 - 0: 2018 - 12 - 22 - 0',
'Refer': 'g'
}
url = 'https://s.weibo.com/weibo/%25E5%25AE%259D%25E5%25A6%2588%25E7%25BE%25A4?' + urlencode(data)
wangzhi = requests.get(url,headers = headers)
return wangzhi.text
def jiexi(html):
doc = pq(html)
item = doc('.m4 li')
for i in item.items():
print(i('img').attr('src'))
def main():
html = dizhi()
print(html)
jiexi(html)
if __name__ == '__main__':
main()
 |
1
jugelizi 2018-12-23 12:02:38 +08:00
是不是异步加载的数据
|
 |
2
zhangsen1992 2018-12-23 12:06:46 +08:00  1
js
|
 |
3
wly19960911 2018-12-23 12:09:19 +08:00 via Android
微博为什么抓页面啊，不是有 API 吗
|
 |
4
helsonxiao 2018-12-23 13:41:52 +08:00 via Android
试试微博移动端的 api，限制少，可以拿来学习
|
 |
5
meiyoumingzi6 2018-12-23 13:45:09 +08:00 via Android 3
|
 |
6
locoz 2018-12-24 16:16:12 +08:00
你在浏览器里看到的是经过 JS 渲染的，想看到和你自己写的程序去请求拿到的 response 完全一样的内容，请使用 postman 之类的工具。
|
Select Language