New GitHub Miner

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

New GitHub Miner

L0 Member

I just wanted to share that I have created and published a miner for github.com address space. It uses the github meta api endpoint and offers miners for each category of github's address space. These are git, hooks, pages, and importer currently.

 

You can find it here: https://github.com/lampwins/github-miner

 

Just follow the instructions for adding an extension to MineMeld.

2 REPLIES 2

L5 Sessionator

Hi @andersonjd,

 

congratulations for such a nice extension and thanks for sharing it with the community.

 

I just want to let you know that you can also mine that API using the SimpleJSON class (standard library). In other words, just providing a new prototype without coding anything.

 

I created the article  Using MineMeld to extract indicators from a generic API  a few weeks ago when, probably, you had already started your project. In the article I described a "generic API" as anything that conforms to:

  • The transport must be HTTP/HTTPS
  • None or basic authentication (user + password)
  • Single transaction (one call retrieves the whole indicator list – no pagination)
  • Indicators are provided in plain, html, csv or json format.

As the https://api.github.com/meta API fits into it I thought it would be a good example to describe here how a SimpleJSON prototype to mine that would look like.

 

Here it is: (prototype for minemeld.ft.json.SimpleJSON class)

 

age_out:
    default: null
    interval: 257
    sudden_death: true
attributes:
    confidence: 100
    share_level: green
    type: IPv4
extractor: '[hooks[].{indicator:@,meta:''hooks''},git[].{indicator:@,meta:''git''},pages[].{indicator:@,meta:''pages''},importer[].{indicator:@,meta:''importer''}][]'
fields:
- meta
source_name: git
url: https://api.github.com/meta

As of today, the following is the feed produced by a miner node using above prototype. Notice the "json-meta" value you can use in input filters if needed.

 

[
  {
    "indicator": "13.229.188.59/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "13.250.177.223/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "140.82.112.0/20",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "18.194.104.89/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "18.195.85.27/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "185.199.108.0/22",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "185.199.108.153/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "pages",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "185.199.109.153/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "pages",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "185.199.110.153/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "pages",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "185.199.111.153/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "pages",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "192.30.252.0/22",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "192.30.252.153/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "pages",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "192.30.252.154/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "pages",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "23.20.92.3",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "importer",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "35.159.8.160/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "52.74.223.119/32",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "git",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "54.166.52.62",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "importer",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  },
  {
    "indicator": "54.87.5.173",
    "value": {
      "sources": [
        "git"
      ],
      "confidence": 100,
      "json_meta": "importer",
      "first_seen": 1536308190282,
      "type": "IPv4",
      "share_level": "green",
      "last_seen": 1536308190282
    }
  }
]

 

Hi,@xhoms

Thanks for this article. Works perfectly!

  • 6207 Views
  • 2 replies
  • 1 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!