Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

Getting this error when scraping JSON with scrapy: Spider must return request, item, or None, got 'str'

I am trying to get a json field with key "longName" with scrapy but I am receiving the error: "Spider must return request, item, or None, got ‘str’".

The JSON I’m trying to scrape looks something like this:

{
   "id":5355,
   "code":9594,
   "longName":"Coolio Note, Series 1164"
}

This is my code:

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

import scrapy
import json

class NotesSpider(scrapy.Spider):
    name = 'notes'
    allowed_domains = ['blahblahblah.com']
    start_urls = ['https://blahblahblah.com/api/123']

    def parse(self, response):
        data = json.loads(response.body)
        yield from data['longName']

I get the above error when I run "scrapy crawl notes" in prompt. Anyone can point me in the right direction?

>Solution :

If you only want longName modifying your parse method like this should do the trick:

    def parse(self, response):
        data = json.loads(response.body)
        yield {"longName": data["longName"]}
Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading