JSON Parsing Issue about mihari

[ninoseki]

I cannot reproduce the error.
The following two commands work in my env.

mihari shodan 'ssl:"MetasploitSelfSignedCA" http.favicon.hash:"-127886975"'
mihari shodan 'hash:-1564315793'

[ssnkhan]

How odd.

The queries work providing there is not a result which contains something that breaks the parser. When that happens, none of the results (for that query) get pushed into the db. I had the same issue today:

/usr/lib/ruby/2.7.0/json/common.rb:156:in `parse'
/usr/lib/ruby/2.7.0/json/common.rb:156:in `parse'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:45:in `block in request'
/usr/lib/ruby/2.7.0/net/http.rb:933:in `start'
/usr/lib/ruby/2.7.0/net/http.rb:606:in `start'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:36:in `request'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/base.rb:56:in `get'
/var/lib/gems/2.7.0/gems/shodanx-0.2.1/lib/shodan/clients/host.rb:36:in `search'
/var/lib/gems/2.7.0/gems/mihari-1.5.0/lib/mihari/analyzers/shodan.rb:44:in `search_with_page'
/var/lib/gems/2.7.0/gems/mihari-1.5.0/lib/mihari/analyzers/shodan.rb:54:in `block in search'

This is the IP causing the issue today: https://beta.shodan.io/host/180.215.106.226

Is there an equivalent of on error resume next or something similar? So if the Metasploit query say has 100 results, if one triggers an error, none of the remaining 99 are added to the DB.

Really appreciate your help.

[ninoseki]

I cannot fix the issue which cannot reproduce

[ssnkhan]

I have done a fresh install of Ruby 3.0 on a fresh Ubuntu machine, and seem to keep running into this issue. I tried updating cli.rb with :quirks_mode => true (i.e. json.parse (input, :quirks_mode => true) but that falls over completely. According to Google, the error is caused by invalid JSON, in this instance being returned by the Shodan API.

Is there anything I can provide with regard to debugging that might help identify the root cause? I am wondering whether this issue happens if there are many results (some searches return 50+ results).

Thanks again for all your help!

[ssnkhan]

Just had another look into the log file that I have been using to capture the errors. Looks like the same parse error is generated when the API returns a api.shodan.io | 502: Bad gateway error. What follows is the Cloudflare interstitial (HTML).

I wonder whether the size of the response could also be at play here; the other JSON errors appear in the log file and abruptly cut off part way through the http.favicon_hash key/value, which does contain numerous characters.

[ninoseki]

@ssnkhan my understandings are:

• Shodan API returns an invalid JSON
• Mihari fails to parse a response from the API (because its format is invalid)

Are they right?
If so I think it's good to claim the issue to Shodan. 🤔

[ninoseki]

Also, please provide the actual command you ran.
It will make me easier to find a clue from your complaints.

[ssnkhan]

Thanks @ninoseki. The query is:

mihari shodan 'ssl:"MetasploitSelfSignedCA" http.favicon.hash:-127886975 -http.title:"Bam! Application Error"' --title "Metasploit C2" --tags "C2"

This query, and maybe 20 others, are ran via a cronjob. Looking at the result on Shodan (for the specific error in my log file) for the host https://beta.shodan.io/host/180.215.106.226/raw, the JSON is complete, but you will note that the response is very lengthy.

Having done some more Googling, I suspect the issue may also be related to throttling by Shodan. Let me add some sleep 5 into my script and see if the issue persists.

[ssnkhan]

Update: sprinkling sleep 5 has had no effect, and the same query continues to generate the above errors (not always, but reasonably consistently).

[ninoseki]

Could you report the issue to Shodan, please?
The issue is caused by the error on the Shoda API side.
There is nothing I can do it with.

[ssnkhan]

Thanks @ninoseki — I’ll raise the issue with Shodan.

However, could error handling be improved so that invalid JSON doesn’t interrupt the importing of other valid IOCs as part of the same query?

[ninoseki]

However, could error handling be improved so that invalid JSON doesn’t interrupt the importing of other valid IOCs as part of the same query?

Okey, it makese sense to me.
I will work on that.