某网站的图片：
通过“https://photo.fengniao.com/#p=4”（人像）进入某一主题。
显示的是几十张缩略的小图片以及相应的跳转地址，点击小图片后获取大图片。
想获取小图片背后的大图片，如果通过串行方法依次访问大图链接后保存，会非常耗时。
1，使用多线程获取图片
import requestsfrom lxml import etreefrom concurrent.futures import ThreadPoolExecutorfrom functools import partialdef get_paths(path, regex, code): ???""" ???:param path: 网页 ???:param regex: 解析规则 ???:param code: 编码 ???:return: 根据解析规则，解析网页后返回内容列表 ???""" ???resp = requests.get(path) ???if resp.status_code == 200: ???????content = resp.content.decode(code) ???????select = etree.HTML(content) ???????paths = select.xpath(regex) ???????return pathsdef save_pic(path, pic_name, directory): ???""" ???:param pic_name: 保存的图片名称 ???:param path: 图片的地址 ???:param directory: 保存的图片目录 ???""" ???resp = requests.get(path, stream=True) ???if resp.status_code == 200: ???????with open(‘{}/{}.jpg‘.format(directory, pic_name), ‘wb‘) as f: ???????????f.write(resp.content)if __name__ == ‘__main__‘: ???paths = get_paths(‘https://photo.fengniao.com/#p=4‘, ‘//a[@class="pic"]/@href‘, ‘utf-8‘) ???paths = [‘https://photo.fengniao.com/‘ + p for p in paths] ???# 获取所有大图片路径 ???p = partial(get_paths, regex=‘//img[@class="picBig"]/@src‘, code=‘utf-8‘) ?# 冻结规则和编码 ???with ThreadPoolExecutor() as excutor: ???????res = excutor.map(p, paths) ???big_paths = [i[0] for i in res] ?# 拿到所有图片的路径 ???# 保存图片 ???p = partial(save_pic, directory=‘fn_pics‘) ??# 冻结保存目录 ???with ThreadPoolExecutor() as excutor: ???????res = excutor.map(p, range(len(big_paths)), big_paths) ???[r for r in res] ?# res是个迭代器，需要遍历触发并发爬取网站图片
原文地址：https://www.cnblogs.com/guxh/p/10351655.html