Miner shows 422 Unprocessable Entity

[otto38dd]

Hi,  I am trying to configure a miner that downlods a stream of IP addresses via HTTPS request.  Data stream looks like this

1.1.1.1

2.2.2.2

2.2.2.3

3.3.3.3

etc.

I created the following protype

  NSFOCUS_ip-v2:
        class: minemeld.ft.http.HttpFT
        config:
            attributes:
                NS-NTI-KEY: *****************
                REPUTATIONTYPE: ip
                TIMETYPE: week
                confidence: 80
            source_name: nsfocus_ip
            url: https://host.server.com/api/v1/reputation/feedDownload/
            verify_cert: false
        description: Detailed feed of IPs classified in different categories. You
            need a valid API to access this feed.
        development_status: EXPERIMENTAL
        indicator_types:
        - IPv4
        node_type: miner
        tags:
        - OSINT
        - Confidence High

 Created a miner from the prototype.  When the miner runs I get a 422 Unprocessable Entity error.

Engine log shows

2018-08-25T22:11:27 (26943)basepoller._poll ERROR: Exception in polling loop for nsfocus-ip: 422 Client Error: UNPROCESSABLE ENTITY
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 721, in _poll
performed = self._polling_loop()
File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 571, in _polling_loop
iterator = self._build_iterator(now)

File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/minemeld/ft/http.py", line 205, in _build_iterator
r.raise_for_status()
File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/requests/models.py", line 851, in raise_for_status
raise HTTPError(http_error_msg, response=self)
HTTPError: 422 Client Error: UNPROCESSABLE ENTITY

Since documentation on error messages are a bit sparse I am not sure why the poller or models are unhappy.  Is there a way to get debug info to see what is happening?

In case anyone asks, verify_cert: false is there because the server has a certificate chain issue.  Using the above in curl works correctly.

Thanks.

[xhoms]

Hi @otto38dd,

as per https://www.keycdn.com/support/422-unprocessable-entity/, error 422 seems to be generated by the server when the requests syntax is incorrect.

You could try to retrieve the content from the OS hosting MineMeld using the curl tool (curl -v <url>) to get insights on the request.

[otto38dd]

HI Xhoms,

That is one of my issues.  How can I see what curl command is actually created within Minemeld?  I do not see any log entry that displays that.  The standard curl request I normally use has no issue so I am sure that I do not have the prototype configured correctly to create the curl.

This is my standard curl.

curl -s -D /tmp/dump-header.txt -o /tmp/curl-out.tgz -H 'NS-NTI-KEY:**************' -H 'REPUTATIONTYPE:file' -H 'TIMETYPE:month' 'https://host.server.com/api/v1/reputation/feedDownload/'
HTTP/1.1 200 OK
Server: nginx/1.10.2
Date: Fri, 31 Aug 2018 01:50:26 GMT
Content-Type: application/octet-stream
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept, Cookie
Allow: POST, OPTIONS, GET
Content-Disposition: attachment;filename=20180831-file-month.tar.gz
Set-Cookie: sessionid=yrzqaml43x6ygnhuxdu0cr5r89apzelf; expires=Fri, 31-Aug-2018 02:50:02 GMT; httponly; Max-Age=3600; Path=/
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=31536000; includeSubdomains;
Access-Control-Allow-Origin: host.server.com
Access-Control-Allow-Headers: X-Requested-With
Access-Control-Allow-Methods: GET,POST,OPTIONS
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=31536000; includeSubdomains;
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: X-Requested-With
Access-Control-Allow-Methods: GET,POST,OPTIONS

received output file: 20180831-file-month

Thanks,

otto38dd

[xhoms]

Hi @otto38dd,

looks like the feed you're trying to "mine" is providing a "tgz" file instead of a HTML, JSON, CSV or plain TXT content:

Content-Type: application/octet-stream
Content-Disposition: attachment;filename=20180831-file-month.tar.gz

The content provided by the feed should be any of the following:

Content-Type: text/plain
Content-Type: text/html
Content-Type: text/csv
Content-Type: application/json

General purpose "miner" classess (HttpFP, CSVFT and SimpleJSON) are "streaming processors". They extract the indicators while the feed content is being parsed. The easiest way to achieve your goal is to implement a CGI script in the WEB server hosting the feed to uncompress the tgz content (i.e. zcat). If that's not possible, then you'll need to create a new miner class that 1) downloads the ".tgz", 2) uncompresses the content and 3) parses the result to extract the indicators.