New GitHub Miner

andersonjd — Fri, 07 Sep 2018 06:31:26 GMT

I just wanted to share that I have created and published a miner for github.com address space. It uses the github meta api endpoint and offers miners for each category of github's address space. These are git, hooks, pages, and importer currently.

You can find it here: https://github.com/lampwins/github-miner

Just follow the instructions for adding an extension to MineMeld.

Re: New GitHub Miner

xhoms — Fri, 07 Sep 2018 08:35:59 GMT

Hi @andersonjd,

congratulations for such a nice extension and thanks for sharing it with the community.

I just want to let you know that you can also mine that API using the SimpleJSON class (standard library). In other words, just providing a new prototype without coding anything.

I created the article Using MineMeld to extract indicators from a generic API a few weeks ago when, probably, you had already started your project. In the article I described a "generic API" as anything that conforms to:

The transport must be HTTP/HTTPS
None or basic authentication (user + password)
Single transaction (one call retrieves the whole indicator list – no pagination)
Indicators are provided in plain, html, csv or json format.

As the https://api.github.com/meta API fits into it I thought it would be a good example to describe here how a SimpleJSON prototype to mine that would look like.

Here it is: (prototype for minemeld.ft.json.SimpleJSON class)

age_out:
    default: null
    interval: 257
    sudden_death: true
attributes:
    confidence: 100
    share_level: green
    type: IPv4
extractor: '[hooks[].{indicator:@,meta:''hooks''},git[].{indicator:@,meta:''git''},pages[].{indicator:@,meta:''pages''},importer[].{indicator:@,meta:''importer''}][]'
fields:
- meta
source_name: git
url: https://api.github.com/meta

As of today, the following is the feed produced by a miner node using above prototype. Notice the "json-meta" value you can use in input filters if needed.

[
  {
    "indicator": "13.229.188.59/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "13.250.177.223/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "140.82.112.0/20",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "18.194.104.89/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "18.195.85.27/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "185.199.108.0/22",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "185.199.108.153/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "pages",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "185.199.109.153/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "pages",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "185.199.110.153/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "pages",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "185.199.111.153/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "pages",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "192.30.252.0/22",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "192.30.252.153/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "pages",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "192.30.252.154/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "pages",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "23.20.92.3",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "importer",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "35.159.8.160/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "52.74.223.119/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "54.166.52.62",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "importer",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "54.87.5.173",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "importer",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  }
]

Re: New GitHub Miner

fwmike — Thu, 19 Sep 2019 18:26:40 GMT

Hi,@xhoms

Thanks for this article. Works perfectly!

topic Re: New GitHub Miner in General Topics

New GitHub Miner

Re: New GitHub Miner

Re: New GitHub Miner