Miner shows 422 Unprocessable Entity

Reply
L0 Member

Miner shows 422 Unprocessable Entity

 

 

 

 

Hi,  I am trying to configure a miner that downlods a stream of IP addresses via HTTPS request.  Data stream looks like this

1.1.1.1

2.2.2.2

2.2.2.3

3.3.3.3

etc.

 

I created the following protype

 

  NSFOCUS_ip-v2:
        class: minemeld.ft.http.HttpFT
        config:
            attributes:
                NS-NTI-KEY: *****************
                REPUTATIONTYPE: ip
                TIMETYPE: week
                confidence: 80
            source_name: nsfocus_ip
            url: https://host.server.com/api/v1/reputation/feedDownload/
            verify_cert: false
        description: Detailed feed of IPs classified in different categories. You
            need a valid API to access this feed.
        development_status: EXPERIMENTAL
        indicator_types:
        - IPv4
        node_type: miner
        tags:
        - OSINT
        - Confidence High

 Created a miner from the prototype.  When the miner runs I get a 422 Unprocessable Entity error.

 

Engine log shows

2018-08-25T22:11:27 (26943)basepoller._poll ERROR: Exception in polling loop for nsfocus-ip: 422 Client Error: UNPROCESSABLE ENTITY
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 721, in _poll
performed = self._polling_loop()
File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 571, in _polling_loop
iterator = self._build_iterator(now)

File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/minemeld/ft/http.py", line 205, in _build_iterator
r.raise_for_status()
File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/requests/models.py", line 851, in raise_for_status
raise HTTPError(http_error_msg, response=self)
HTTPError: 422 Client Error: UNPROCESSABLE ENTITY

 

Since documentation on error messages are a bit sparse I am not sure why the poller or models are unhappy.  Is there a way to get debug info to see what is happening?

 

In case anyone asks, verify_cert: false is there because the server has a certificate chain issue.  Using the above in curl works correctly.

 

Thanks.

 

 

L5 Sessionator

Re: Miner shows 422 Unprocessable Entity

Hi @otto38dd,

 

as per https://www.keycdn.com/support/422-unprocessable-entity/, error 422 seems to be generated by the server when the requests syntax is incorrect.

 

You could try to retrieve the content from the OS hosting MineMeld using the curl tool (curl -v <url>) to get insights on the request.

L0 Member

Re: Miner shows 422 Unprocessable Entity

HI Xhoms,

 

That is one of my issues.  How can I see what curl command is actually created within Minemeld?  I do not see any log entry that displays that.  The standard curl request I normally use has no issue so I am sure that I do not have the prototype configured correctly to create the curl.

 

This is my standard curl.

 


curl -s -D /tmp/dump-header.txt -o /tmp/curl-out.tgz -H 'NS-NTI-KEY:**************' -H 'REPUTATIONTYPE:file' -H 'TIMETYPE:month' 'https://host.server.com/api/v1/reputation/feedDownload/'
HTTP/1.1 200 OK
Server: nginx/1.10.2
Date: Fri, 31 Aug 2018 01:50:26 GMT
Content-Type: application/octet-stream
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept, Cookie
Allow: POST, OPTIONS, GET
Content-Disposition: attachment;filename=20180831-file-month.tar.gz
Set-Cookie: sessionid=yrzqaml43x6ygnhuxdu0cr5r89apzelf; expires=Fri, 31-Aug-2018 02:50:02 GMT; httponly; Max-Age=3600; Path=/
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=31536000; includeSubdomains;
Access-Control-Allow-Origin: host.server.com
Access-Control-Allow-Headers: X-Requested-With
Access-Control-Allow-Methods: GET,POST,OPTIONS
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=31536000; includeSubdomains;
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: X-Requested-With
Access-Control-Allow-Methods: GET,POST,OPTIONS


received output file: 20180831-file-month

 

Thanks,

 

otto38dd

 

L5 Sessionator

Re: Miner shows 422 Unprocessable Entity

Hi @otto38dd,

 

looks like the feed you're trying to "mine" is providing a "tgz" file instead of a HTML, JSON, CSV or plain TXT content:

 

Content-Type: application/octet-stream
Content-Disposition: attachment;filename=20180831-file-month.tar.gz

The content provided by the feed should be any of the following:

 

Content-Type: text/plain
Content-Type: text/html
Content-Type: text/csv
Content-Type: application/json

 

General purpose "miner" classess (HttpFP, CSVFT and SimpleJSON) are "streaming processors". They extract the indicators while the feed content is being parsed. The easiest way to achieve your goal is to implement a CGI script in the WEB server hosting the feed to uncompress the tgz content (i.e. zcat). If that's not possible, then you'll need to create a new miner class that 1) downloads the ".tgz", 2) uncompresses the content and 3) parses the result to extract the indicators.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!