Comments (4)
@maintain99
你好,這是 Scrapy 新版提出的改進。簡單來說,在 self.spider
裡面直接寫入你要呼叫的 spider 就行了,不需要傳參數進去,一來方便,二來安全。
from scrapy-redis.
@LuckyPigeon 我该怎么改能详细说一下吗,谢谢
from scrapy-redis.
@maintain99
我需要你的 code 才能詳細解說,但照理說你只要在 parse 內或任何你呼叫 scrapy 的函數內指定 self.spider = ExampleSpider
就行了。
但如果你對 spider 還不熟悉,建議你不要管 warning 了,還是能過的
from scrapy-redis.
有警告看着很不爽
`
import scrapy
from scrapy_redis.spiders import RedisSpider
class ShangZhiSpider(RedisSpider):
name= 'shangzhi'
allowed_domains = ['che168.com']
redis_key = 'chaosui_urls'
def parse(self, response,**kwargs):
lis=response.xpath("//ul[@class='viewlist_ul']/li/a/@href").extract()
for lj in lis:
herf=response.urljoin(lj)
yield scrapy.Request(
url=herf,
callback=self.jiexi
)
print(herf)
lst=response.xpath('//div[@class="page fn-clear"]/a/@href').extract()[1:]
for url in lst:
urll=response.joinurl(url)
yield scrapy.Request(
url=urll,
callback=self.parse
)
def jiexi(self, resp):
titer=resp.xpath('//h3[@class="car-brand-name"]/text()').extract()
print(titer[0].strip())
print(resp.url)
`
from scrapy-redis.
Related Issues (20)
- ImportError: cannot import name 'Iterable' from 'collections' HOT 2
- Is there a way to stop spider check duplicate with redis ? HOT 5
- [docs] Chinese documentation
- [style] Fix pylint style
- Redis data persistence HOT 4
- Scrapy 2.6.1 Unsupport function make_requests_from_url to use HOT 5
- error object has no attribute 'make_requests_from_url' HOT 2
- [spiders] remove duplicate check setting types
- why I can not see request record in redis HOT 2
- 你好,大佬,请问一个问题,我用scrapy-redis执行爬虫的时候,设置最大并发是20,但是怎么感觉像创建20个队列一样,上一个20执行完之后,下一个20再执行,大佬,您能为我解答一下吗,感激不尽 HOT 2
- How to use scrapy-redis if I'm using start_requests() instead of start_urls in my spider? HOT 3
- 日志报警:String request is deprecated
- make_request_from_data implementation in RedisMixin HOT 2
- Cleanup requirements HOT 1
- Playwright? HOT 2
- Scrapy 2.8.0 deprecated function scrapy.utils.request.request_fingerprint() warning HOT 1
- Is there a planned support for Python3.11? HOT 5
- [dev] Add Type annotations
- Add Type annotations pep-0483
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scrapy-redis.