XSIAM HTTP Log Collector Testing

[J.Huchtktter]

Hello everyone,

We are currently considering to us a HTTP Log Collector.

However, before this can be started, I wanted to test the collection. 

Now, in theory, this should not be hard, due to the examples provided after setting up the integration.

Nonetheless, either the Python Example is faulty in some way, or I am being exceedingly stupid.

Currently, my code looks like this, very close to the provided example:

def test_http_collector(api_key: str) -> requests.Response:
    headers = {
        "Authorization": api_key,
        "Content-Type": "text/plain"
    }
    # Note: the logs must be separated by a new line
    body = "{'example1': 'test', 'timestamp': 1609100113039}\n{'example2': [12321,546456,45687,1]}"
    res = requests.post(url="https://api-<name>.xdr.de.paloaltonetworks.com/logs/v1/event",
                        headers=headers,
                        data=body,
                        timeout=600
                        )
    return res

I've only added a timeout and changed the linebreak (which would not add a newline) to a \n.

This consistently seems to return HTTP 500 for me.

The best I've managed in trying multiple ways for hours is HTTP 400 and I cannot even replicate it right now.

Does someone who is running a working HTTP log collector have any idea?

Or maybe a tested and proven, minimal working example?

Is there anywhere I can check what actually happened in XSIAM?

[rpasca · Accepted solution]

Hi there,

I tested the example in the documentation, and I could not get it to work until I fixed the single/double quotes.

Please test by replacing them as such:

body = '{"example": "test", "timestamp": 1609100113039}\n{"example2": [12321,546456,45687,1]}'

Let me know if this helps.

BR

Raúl

[J.Huchtktter]

Thank you!

This is what I must have done once, because it at least produces a 400 - Bad Request.

Which is further then I got, but I still do not know why this is happening.

[J.Huchtktter · Accepted solution]

A bit of testing has turned out a few more things, and likely the cause of the specific HTTP 400:

While Raúl was correct about the quotes being an issue with Python, the HTTP 400 error seems to arise whenever a HTTP collector instance is changed - e.g. the input type is changed from RAW to JSON, or the compression is changed from Gzip to uncompressed or ...

As this has cost me a lot of nerves and testing, here are some of my findings

• Set up the HTTP based Collector - set it up in the way you want it to run. If you ever have to change anything about it, create a new one and redistribute the API key
• JSON-collectors do not actually seem to care if the application type is set to application/json or text/plain 
• Sending uncompressed data to a gzip collector works, too
• Multiple JSON log lines must be sent as on json object per line, separated by line breaks

Working Python example function:

def test_http_collector(api_key: str, collector_url: str) -> requests.Response:
    headers = {"Authorization": api_key, "Content-Type": "application/json"}
    # Note: the logs must be separated by a new line
    msg_1 = '{"msg": "This is my test!", "severity": "High", "id": "EJhn90nsdui2", "duration": 432, "timestamp": 1609100113039}'
    msg_2 = '{"msg": "I am also a test", "severity": "Low", "id": "kdrjhWo834jhz", "duration": 20, "timestamp": 1609100113052}'
    body = f"{msg_1}\n{msg_2}"

    return requests.post(
        url=collector_url,
        headers=headers,
        data=body,
        timeout=600
    )