09-06-2018 11:31 PM
I just wanted to share that I have created and published a miner for github.com address space. It uses the github meta api endpoint and offers miners for each category of github's address space. These are git, hooks, pages, and importer currently.
You can find it here: https://github.com/lampwins/github-miner
Just follow the instructions for adding an extension to MineMeld.
09-07-2018 01:34 AM - edited 09-07-2018 01:35 AM
Hi @andersonjd,
congratulations for such a nice extension and thanks for sharing it with the community.
I just want to let you know that you can also mine that API using the SimpleJSON class (standard library). In other words, just providing a new prototype without coding anything.
I created the article Using MineMeld to extract indicators from a generic API a few weeks ago when, probably, you had already started your project. In the article I described a "generic API" as anything that conforms to:
As the https://api.github.com/meta API fits into it I thought it would be a good example to describe here how a SimpleJSON prototype to mine that would look like.
Here it is: (prototype for minemeld.ft.json.SimpleJSON class)
age_out:
default: null
interval: 257
sudden_death: true
attributes:
confidence: 100
share_level: green
type: IPv4
extractor: '[hooks[].{indicator:@,meta:''hooks''},git[].{indicator:@,meta:''git''},pages[].{indicator:@,meta:''pages''},importer[].{indicator:@,meta:''importer''}][]'
fields:
- meta
source_name: git
url: https://api.github.com/meta
As of today, the following is the feed produced by a miner node using above prototype. Notice the "json-meta" value you can use in input filters if needed.
[
{
"indicator": "13.229.188.59/32",
"value": {
"sources": [
"git"
],
"confidence": 100,
"json_meta": "git",
"first_seen": 1536308190282,
"type": "IPv4",
"share_level": "green",
"last_seen": 1536308190282
}
},
{
"indicator": "13.250.177.223/32",
"value": {
"sources": [
"git"
],
"confidence": 100,
"json_meta": "git",
"first_seen": 1536308190282,
"type": "IPv4",
"share_level": "green",
"last_seen": 1536308190282
}
},
{
"indicator": "140.82.112.0/20",
"value": {
"sources": [
"git"
],
"confidence": 100,
"json_meta": "git",
"first_seen": 1536308190282,
"type": "IPv4",
"share_level": "green",
"last_seen": 1536308190282
}
},
{
"indicator": "18.194.104.89/32",
"value": {
"sources": [
"git"
],
"confidence": 100,
"json_meta": "git",
"first_seen": 1536308190282,
"type": "IPv4",
"share_level": "green",
"last_seen": 1536308190282
}
},
{
"indicator": "18.195.85.27/32",
"value": {
"sources": [
"git"
],
"confidence": 100,
"json_meta": "git",
"first_seen": 1536308190282,
"type": "IPv4",
"share_level": "green",
"last_seen": 1536308190282
}
},
{
"indicator": "185.199.108.0/22",
"value": {
"sources": [
"git"
],
"confidence": 100,
"json_meta": "git",
"first_seen": 1536308190282,
"type": "IPv4",
"share_level": "green",
"last_seen": 1536308190282
}
},
{
"indicator": "185.199.108.153/32",
"value": {
"sources": [
"git"
],
"confidence": 100,
"json_meta": "pages",
"first_seen": 1536308190282,
"type": "IPv4",
"share_level": "green",
"last_seen": 1536308190282
}
},
{
"indicator": "185.199.109.153/32",
"value": {
"sources": [
"git"
],
"confidence": 100,
"json_meta": "pages",
"first_seen": 1536308190282,
"type": "IPv4",
"share_level": "green",
"last_seen": 1536308190282
}
},
{
"indicator": "185.199.110.153/32",
"value": {
"sources": [
"git"
],
"confidence": 100,
"json_meta": "pages",
"first_seen": 1536308190282,
"type": "IPv4",
"share_level": "green",
"last_seen": 1536308190282
}
},
{
"indicator": "185.199.111.153/32",
"value": {
"sources": [
"git"
],
"confidence": 100,
"json_meta": "pages",
"first_seen": 1536308190282,
"type": "IPv4",
"share_level": "green",
"last_seen": 1536308190282
}
},
{
"indicator": "192.30.252.0/22",
"value": {
"sources": [
"git"
],
"confidence": 100,
"json_meta": "git",
"first_seen": 1536308190282,
"type": "IPv4",
"share_level": "green",
"last_seen": 1536308190282
}
},
{
"indicator": "192.30.252.153/32",
"value": {
"sources": [
"git"
],
"confidence": 100,
"json_meta": "pages",
"first_seen": 1536308190282,
"type": "IPv4",
"share_level": "green",
"last_seen": 1536308190282
}
},
{
"indicator": "192.30.252.154/32",
"value": {
"sources": [
"git"
],
"confidence": 100,
"json_meta": "pages",
"first_seen": 1536308190282,
"type": "IPv4",
"share_level": "green",
"last_seen": 1536308190282
}
},
{
"indicator": "23.20.92.3",
"value": {
"sources": [
"git"
],
"confidence": 100,
"json_meta": "importer",
"first_seen": 1536308190282,
"type": "IPv4",
"share_level": "green",
"last_seen": 1536308190282
}
},
{
"indicator": "35.159.8.160/32",
"value": {
"sources": [
"git"
],
"confidence": 100,
"json_meta": "git",
"first_seen": 1536308190282,
"type": "IPv4",
"share_level": "green",
"last_seen": 1536308190282
}
},
{
"indicator": "52.74.223.119/32",
"value": {
"sources": [
"git"
],
"confidence": 100,
"json_meta": "git",
"first_seen": 1536308190282,
"type": "IPv4",
"share_level": "green",
"last_seen": 1536308190282
}
},
{
"indicator": "54.166.52.62",
"value": {
"sources": [
"git"
],
"confidence": 100,
"json_meta": "importer",
"first_seen": 1536308190282,
"type": "IPv4",
"share_level": "green",
"last_seen": 1536308190282
}
},
{
"indicator": "54.87.5.173",
"value": {
"sources": [
"git"
],
"confidence": 100,
"json_meta": "importer",
"first_seen": 1536308190282,
"type": "IPv4",
"share_level": "green",
"last_seen": 1536308190282
}
}
]
09-19-2019 11:26 AM
Hi,@xhoms
Thanks for this article. Works perfectly!
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!
