Calling Scrapy from another file without threading-open source projects scrapy/scrapy

Liαrεz

The first thing I would say to you is when you’re executing Scrapy from external file the loglevel is set to INFO,you should change it to DEBUG to see what’s happening if your code doesn’t work

you should change the line:

 log.start()

for:

log.start(loglevel=log.DEBUG)

To store everything in the log and generate a text file (for debugging purposes) you can do:

log.start(logfile="file.log", loglevel=log.DEBUG, crawler=crawler, logstdout=False)

About the signals issue with the log level changed to DEBUG maybe you can see some output that can help you to fix it, you can try to put your script into the Scrapy Project folder to see if still crashes.

If you change the line:

crawler.signals.connect(reactor.stop, signal=signals.spider_closed)

for:

dispatcher.connect(reactor.stop, signals.spider_closed)

What does it say ?

Depending on your Scrapy version it may be deprecated