Miner shows 422 Unprocessable Entity

L0 Member

Miner shows 422 Unprocessable Entity





Hi,  I am trying to configure a miner that downlods a stream of IP addresses via HTTPS request.  Data stream looks like this



I created the following protype


        class: minemeld.ft.http.HttpFT
                NS-NTI-KEY: *****************
                REPUTATIONTYPE: ip
                TIMETYPE: week
                confidence: 80
            source_name: nsfocus_ip
            url: https://host.server.com/api/v1/reputation/feedDownload/
            verify_cert: false
        description: Detailed feed of IPs classified in different categories. You
            need a valid API to access this feed.
        development_status: EXPERIMENTAL
        - IPv4
        node_type: miner
        - OSINT
        - Confidence High

 Created a miner from the prototype.  When the miner runs I get a 422 Unprocessable Entity error.


Engine log shows

2018-08-25T22:11:27 (26943)basepoller._poll ERROR: Exception in polling loop for nsfocus-ip: 422 Client Error: UNPROCESSABLE ENTITY
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 721, in _poll
performed = self._polling_loop()
File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 571, in _polling_loop
iterator = self._build_iterator(now)

File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/minemeld/ft/http.py", line 205, in _build_iterator
File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/requests/models.py", line 851, in raise_for_status
raise HTTPError(http_error_msg, response=self)


Since documentation on error messages are a bit sparse I am not sure why the poller or models are unhappy.  Is there a way to get debug info to see what is happening?


In case anyone asks, verify_cert: false is there because the server has a certificate chain issue.  Using the above in curl works correctly.





L5 Sessionator

Hi @otto38dd,


as per https://www.keycdn.com/support/422-unprocessable-entity/, error 422 seems to be generated by the server when the requests syntax is incorrect.


You could try to retrieve the content from the OS hosting MineMeld using the curl tool (curl -v <url>) to get insights on the request.

L0 Member

HI Xhoms,


That is one of my issues.  How can I see what curl command is actually created within Minemeld?  I do not see any log entry that displays that.  The standard curl request I normally use has no issue so I am sure that I do not have the prototype configured correctly to create the curl.


This is my standard curl.


curl -s -D /tmp/dump-header.txt -o /tmp/curl-out.tgz -H 'NS-NTI-KEY:**************' -H 'REPUTATIONTYPE:file' -H 'TIMETYPE:month' 'https://host.server.com/api/v1/reputation/feedDownload/'
HTTP/1.1 200 OK
Server: nginx/1.10.2
Date: Fri, 31 Aug 2018 01:50:26 GMT
Content-Type: application/octet-stream
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept, Cookie
Content-Disposition: attachment;filename=20180831-file-month.tar.gz
Set-Cookie: sessionid=yrzqaml43x6ygnhuxdu0cr5r89apzelf; expires=Fri, 31-Aug-2018 02:50:02 GMT; httponly; Max-Age=3600; Path=/
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=31536000; includeSubdomains;
Access-Control-Allow-Origin: host.server.com
Access-Control-Allow-Headers: X-Requested-With
Access-Control-Allow-Methods: GET,POST,OPTIONS
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=31536000; includeSubdomains;
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: X-Requested-With
Access-Control-Allow-Methods: GET,POST,OPTIONS

received output file: 20180831-file-month






L5 Sessionator

Hi @otto38dd,


looks like the feed you're trying to "mine" is providing a "tgz" file instead of a HTML, JSON, CSV or plain TXT content:


Content-Type: application/octet-stream
Content-Disposition: attachment;filename=20180831-file-month.tar.gz

The content provided by the feed should be any of the following:


Content-Type: text/plain
Content-Type: text/html
Content-Type: text/csv
Content-Type: application/json


General purpose "miner" classess (HttpFP, CSVFT and SimpleJSON) are "streaming processors". They extract the indicators while the feed content is being parsed. The easiest way to achieve your goal is to implement a CGI script in the WEB server hosting the feed to uncompress the tgz content (i.e. zcat). If that's not possible, then you'll need to create a new miner class that 1) downloads the ".tgz", 2) uncompresses the content and 3) parses the result to extract the indicators.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!