How to use scrapy-redis if I'm using start_requests() instead of start_urls in my spider? about scrapy-redis

[rmax]

See this method https://github.com/rmax/scrapy-redis/blob/master/src/scrapy_redis/spiders.py#L137

It already supports loading JSON data from redis as start urls.

I think you just need to push json-serialized data ({"url": ..., "meta": {"someid": ...}}) to the <spidername>:start_urls key in your redis instance.

[LuckyPigeon]

@ashwinbala99
Just like @rmax said, just follow the tutorial, then you can send the requests with json-serialized data. Feel free to ask if you have any further questions :)

[ashwinbala99]

Hi, Thank you for responding to my query promptly. I'll certainly reach out if I have any more questions regarding the usage of the library. Regards Ashwin B

…

On Sun, Dec 11, 2022 at 5:00 PM Jeremy Chou ***@***.***> wrote: @ashwinbala99 <https://github.com/ashwinbala99> Just like @rmax <https://github.com/rmax> said, just follow the tutorial <https://github.com/rmax/scrapy-redis/wiki/Introduction>, then you can send the requests with json-serialized data. Feel free to ask if you have any further questions :) — Reply to this email directly, view it on GitHub <#264 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3U7I6Q2TZIYJKZRNTJIBHDWMW3MFANCNFSM6AAAAAASPN6KAI> . You are receiving this because you were mentioned.Message ID: ***@***.***>