New GitHub Miner

Reply
L0 Member

New GitHub Miner

I just wanted to share that I have created and published a miner for github.com address space. It uses the github meta api endpoint and offers miners for each category of github's address space. These are git, hooks, pages, and importer currently.

 

You can find it here: https://github.com/lampwins/github-miner

 

Just follow the instructions for adding an extension to MineMeld.

Highlighted
L5 Sessionator

Re: New GitHub Miner

Hi @andersonjd,

 

congratulations for such a nice extension and thanks for sharing it with the community.

 

I just want to let you know that you can also mine that API using the SimpleJSON class (standard library). In other words, just providing a new prototype without coding anything.

 

I created the article  Using MineMeld to extract indicators from a generic API  a few weeks ago when, probably, you had already started your project. In the article I described a "generic API" as anything that conforms to:

  • The transport must be HTTP/HTTPS
  • None or basic authentication (user + password)
  • Single transaction (one call retrieves the whole indicator list – no pagination)
  • Indicators are provided in plain, html, csv or json format.

As the https://api.github.com/meta API fits into it I thought it would be a good example to describe here how a SimpleJSON prototype to mine that would look like.

 

Here it is: (prototype for minemeld.ft.json.SimpleJSON class)

 

age_out:
    default: null
    interval: 257
    sudden_death: true
attributes:
    confidence: 100
    share_level: green
    type: IPv4
extractor: '[hooks[].{indicator:@,meta:''hooks''},git[].{indicator:@,meta:''git''},pages[].{indicator:@,meta:''pages''},importer[].{indicator:@,meta:''importer''}][]'
fields:
- meta
source_name: git
url: https://api.github.com/meta

As of today, the following is the feed produced by a miner node using above prototype. Notice the "json-meta" value you can use in input filters if needed.

 

[
  {
    "indicator": "13.229.188.59/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "13.250.177.223/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "140.82.112.0/20",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "18.194.104.89/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "18.195.85.27/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "185.199.108.0/22",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "185.199.108.153/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "pages",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "185.199.109.153/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "pages",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "185.199.110.153/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "pages",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "185.199.111.153/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "pages",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "192.30.252.0/22",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "192.30.252.153/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "pages",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "192.30.252.154/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "pages",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "23.20.92.3",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "importer",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "35.159.8.160/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "52.74.223.119/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "54.166.52.62",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "importer",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "54.87.5.173",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "importer",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  }
]

 

L2 Linker

Re: New GitHub Miner

Hi,@xhoms

Thanks for this article. Works perfectly!

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!