https://live.paloaltonetworks.com/t5/general-articles/configuring-nodes/ta-p/77185 <P>The set of config parameters supported by a node depends on the node class. Node configs are stored inside prototypes.<A target="_blank" name="user-content-base-class"></A></P> <P>&nbsp;</P> <H2><A id="user-content-base-class" class="anchor" href="https://github.com/PaloAltoNetworks/minemeld-core/blob/develop/docs/nodeconfig.rst#base-class" target="_blank" rel="noopener"></A>Base class</H2> <P>All nodes support these parameters.</P> <P><A target="_blank" name="user-content-parameters"></A></P> <H2>&nbsp;</H2> <H3><A id="user-content-parameters" class="anchor" href="https://github.com/PaloAltoNetworks/minemeld-core/blob/develop/docs/nodeconfig.rst#parameters" target="_blank" rel="noopener"></A>Parameters</H3> <TABLE frame="void" rules="none"> <TBODY valign="top"> <TR> <TH>infilters:</TH> <TD>inbound <EM>filter set</EM>. Filters to apply to received indicators.</TD> </TR> <TR> <TH>outfilters:</TH> <TD>outbound <EM>filter set</EM>. Filters to apply to transmitted indicators.</TD> </TR> </TBODY> </TABLE> <P><A target="_blank" name="user-content-filter-set"></A></P> <H2>&nbsp;</H2> <H3><A id="user-content-filter-set" class="anchor" href="https://github.com/PaloAltoNetworks/minemeld-core/blob/develop/docs/nodeconfig.rst#filter-set" target="_blank" rel="noopener"></A>Filter set</H3> <P>Each filter set is a list of filters. Filters are checked from top to bottom, the first matching filter is applied and following filters are not checked. Default action is <STRONG>accept</STRONG>. Each filter is a dictionary with 3 keys:</P> <P>&nbsp;</P> <TABLE frame="void" rules="none"> <TBODY valign="top"> <TR> <TH>name:</TH> <TD>name of the filter.</TD> </TR> <TR> <TH>conditions:</TH> <TD>list of boolean expressions to match on the indicator and indicator value.</TD> </TR> <TR> <TH>actions:</TH> <TD>list of actions to apply to the indicator. Currently the only supported actions are <STRONG>accept</STRONG> and <STRONG>drop</STRONG></TD> </TR> </TBODY> </TABLE> <P>&nbsp;</P> <P>In addition to the atttributes in the indicator value, filters can match on 3 special attributes:</P> <TABLE frame="void" rules="none"> <TBODY valign="top"> <TR> <TH>__indicator:</TH> <TD>the indicator itself.</TD> </TR> <TR> <TH>__method:</TH> <TD>the method of the message, <STRONG>update</STRONG> or <STRONG>withdraw</STRONG>.</TD> </TR> <TR> <TH>__origin:</TH> <TD>the name of the node who sent the indicator.</TD> </TR> </TBODY> </TABLE> <P><A target="_blank" name="user-content-condition"></A></P> <P>&nbsp;</P> <H4><A id="user-content-condition" class="anchor" href="https://github.com/PaloAltoNetworks/minemeld-core/blob/develop/docs/nodeconfig.rst#condition" target="_blank" rel="noopener">Condition</A></H4> <P>A condition in the filter is a boolean expression composed by: a JMESPath expression, an operator (&lt;, &lt;=, ==, &gt;=, &gt;, !=) and a value.</P> <P><A target="_blank" name="user-content-example"></A></P> <H3>&nbsp;</H3> <H4><A id="user-content-example" class="anchor" href="https://github.com/PaloAltoNetworks/minemeld-core/blob/develop/docs/nodeconfig.rst#example" target="_blank" rel="noopener">Example</A></H4> <P>Example config in YAML:</P> <PRE>infilters: - name: accept withdraws conditions: - __method == 'withdraw' actions: - accept - name: accept URL conditions: - type == 'URL' actions: - accept - name: drop all actions: - drop outfilters: - name: accept all (default) actions: - accept </PRE> <P><A target="_blank" name="user-content-base-poller-class"></A></P> <H1>&nbsp;</H1> <H2>Base poller class</H2> <P>In addition to <STRONG>Base class</STRONG> config parameters, the base poller class supports the following parameters.</P> <P><A target="_blank" name="user-content-config-parameters"></A></P> <H3>&nbsp;</H3> <H4><A id="user-content-config-parameters" class="anchor" href="https://github.com/PaloAltoNetworks/minemeld-core/blob/develop/docs/nodeconfig.rst#config-parameters" target="_blank" rel="noopener">Config parameters</A></H4> <TABLE frame="void" rules="none"> <TBODY valign="top"> <TR> <TH>source_name:</TH> <TD>name of the source. This is added to the <EM>sources</EM> attribute of the generated indicators. Default: name of the node.</TD> </TR> <TR> <TH>attributes:</TH> <TD>dictionary of attributes for the generated indicators. This dictionary is used as template for the value of the generated indicators. Default: empty</TD> </TR> <TR> <TH>interval:</TH> <TD>polling interval in seconds. Default: 3600.</TD> </TR> <TR> <TH>num_retries:</TH> <TD>how many times the miner should try to reach the source in case of failure. If this number is exceeded, the miner waits until the next polling time to try again. Default: 2</TD> </TR> <TR> <TH>age_out:</TH> <TD>age out policies to apply to the indicators. Default: age out check interval 3600 seconds, sudden death enabled, default age out interval 30 days.</TD> </TR> </TBODY> </TABLE> <P><A target="_blank" name="user-content-age-out-policy"></A></P> <H3>&nbsp;</H3> <H4><A id="user-content-age-out-policy" class="anchor" href="https://github.com/PaloAltoNetworks/minemeld-core/blob/develop/docs/nodeconfig.rst#age-out-policy" target="_blank" rel="noopener">Age out policy</A></H4> <P>Age out policy is described by a dictionary with at least 3 keys:</P> <P>&nbsp;</P> <TABLE frame="void" rules="none"> <TBODY valign="top"> <TR> <TH>interval:</TH> <TD>number of seconds between successive age out checks.</TD> </TR> <TR> <TH>sudden_death:</TH> <TD>boolean, if <EM>true</EM> indicators are immediately aged out when they disappear from the feed.</TD> </TR> <TR> <TH>default:</TH> <TD>age out interval. After this interval an indicator is aged out even if it is still present in the feed. If <EM>null</EM>, no age out interval is applied.</TD> </TR> </TBODY> </TABLE> <P>&nbsp;</P> <P>Additional keys can be used to specify age out interval per indicator <EM>type</EM>.</P> <P><A target="_blank" name="user-content-age-out-interval"></A></P> <P>&nbsp;</P> <H4><A id="user-content-age-out-interval" class="anchor" href="https://github.com/PaloAltoNetworks/minemeld-core/blob/develop/docs/nodeconfig.rst#age-out-interval" target="_blank" rel="noopener">Age out interval</A></H4> <P>Age out intervals have the following format:</P> <PRE>&lt;base attribute&gt;+&lt;interval&gt; </PRE> <P><EM>base attribute</EM> can be <EM>last_seen</EM>, if the age out interval should be calculated based on the last time the indicator was found in the feed, or <EM>first_seen</EM>, if instead the age out interval should be based on the time the indicator was first seen in the feed. If not specified <EM>first_seen</EM> is used.</P> <P><EM>interval</EM> is the length of the interval expressed in seconds. Suffixes <EM>d</EM>, <EM>h</EM> and <EM>m</EM> can be used to specify days, hours or minutes.</P> <P><A target="_blank" name="user-content-id1"></A></P> <P>&nbsp;</P> <H4><A id="user-content-example-1" class="anchor" href="https://github.com/PaloAltoNetworks/minemeld-core/blob/develop/docs/nodeconfig.rst#example-1" target="_blank" rel="noopener">Example</A></H4> <P>Example config in YAML for a feed where indicators should be aged out only when they are removed from the feed:</P> <P>&nbsp;</P> <PRE>source_name: example.persistent_feed interval: 600 age_out: default: null sudden_death: true interval: 300 attributes: type: IPv4 confidence: 100 share_level: green direction: inbound </PRE> <P>&nbsp;</P> <P>Example config in YAML for a feed where indicators are aged out when they disappear from the feed and 30 days after they have seen for the first time in the feed:</P> <P>&nbsp;</P> <PRE>source_name: example.long_running_feed interval: 3600 age_out: default: first_seen+30d sudden_death: true interval: 1800 attributes: type: URL confidence: 50 share_level: green </PRE> <P>&nbsp;</P> <P>Example config in YAML for a feed where indicators are aged 30 days after they have seen for the last time in the feed:</P> <P>&nbsp;</P> <PRE>source_name: example.delta_feed interval: 3600 age_out: default: last_seen+30d sudden_death: false interval: 1800 attributes: type: URL confidence: 50 share_level: green </PRE> <P><A target="_blank" name="user-content-minemeld-ft-http-httpft"></A></P> <H1>&nbsp;</H1> <H2><A id="user-content-minemeldfthttphttpft" class="anchor" href="https://github.com/PaloAltoNetworks/minemeld-core/blob/develop/docs/nodeconfig.rst#minemeldfthttphttpft" target="_blank" rel="noopener">minemeld.ft.http.HttpFT class</A></H2> <P>In addition to <STRONG>Base poller class</STRONG> config parameters, the HttpFT&nbsp;class supports the following parameters:</P> <P><A target="_blank" name="user-content-id2"></A></P> <H3>&nbsp;</H3> <H4><A id="user-content-parameters-1" class="anchor" href="https://github.com/PaloAltoNetworks/minemeld-core/blob/develop/docs/nodeconfig.rst#parameters-1" target="_blank" rel="noopener">Parameters</A></H4> <TABLE frame="void" rules="none"> <TBODY valign="top"> <TR> <TH>url:</TH> <TD>URL of the feed.</TD> </TR> <TR> <TH>polling_timeout:</TH> <TD>timeout of the polling request in seconds. Default: 20</TD> </TR> <TR> <TH>verify_cert:</TH> <TD>boolean, if <EM>true</EM> feed HTTPS server certificate is verified. Default: <EM>true</EM></TD> </TR> <TR> <TH>ignore_regex:</TH> <TD>Python regular expression for lines that should be ignored. Default: <EM>null</EM></TD> </TR> <TR> <TH>indicator:</TH> <TD>an <EM>extraction dictionary</EM> to extract the indicator from the line. If <EM>null</EM>, the text until the first whitespace or newline character is used as indicator. Default: <EM>null</EM></TD> </TR> <TR> <TH>fields:</TH> <TD>a dicionary of <EM>extraction dictionaries</EM> to extract additional attributes from each line. Default: {}</TD> </TR> </TBODY> </TABLE> <P><A target="_blank" name="user-content-extraction-dictionary"></A></P> <H3>&nbsp;</H3> <H4><A id="user-content-extraction-dictionary" class="anchor" href="https://github.com/PaloAltoNetworks/minemeld-core/blob/develop/docs/nodeconfig.rst#extraction-dictionary" target="_blank" rel="noopener">Extraction dictionary</A></H4> <P>Extraction dictionaries contain the following keys:</P> <P>&nbsp;</P> <TABLE frame="void" rules="none"> <TBODY valign="top"> <TR> <TH>regex:</TH> <TD>Python regular expression for searching the text.</TD> </TR> <TR> <TH>transform:</TH> <TD>template to generate the final value from the result of the regular expression. Default: the entire match of the regex is used as extracted value.</TD> </TR> </TBODY> </TABLE> <P>&nbsp;</P> <P>See Python <A href="https://docs.python.org/2/library/re.html" target="_blank" rel="noopener">re</A> module for details about Python regular expressions and templates.</P> <P><A target="_blank" name="user-content-id3"></A></P> <H3>&nbsp;</H3> <H4><A id="user-content-example-2" class="anchor" href="https://github.com/PaloAltoNetworks/minemeld-core/blob/develop/docs/nodeconfig.rst#example-2" target="_blank" rel="noopener">Example</A></H4> <P>Example config in YAML where extraction dictionaries are used to extract the indicator and additional fields:</P> <PRE>url: https://www.dshield.org/block.txt ignore_regex: "[#S].*" indicator: regex: '^([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})\t([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})' transform: '\1-\2' fields: dshield_nattacks: regex: '^.*\t.*\t[0-9]+\t([0-9]+)' transform: '\1' dshield_name: regex: '^.*\t.*\t[0-9]+\t[0-9]+\t([^\t]+)' transform: '\1' dshield_country: regex: '^.*\t.*\t[0-9]+\t[0-9]+\t[^\t]+\t([A-Z]+)' transform: '\1' dshield_email: regex: '^.*\t.*\t[0-9]+\t[0-9]+\t[^\t]+\t[A-Z]+\t(\S+)' transform: '\1' </PRE> <P>&nbsp;</P> <P>Example config in YAML where the text in each line until the first whitespace is used as indicator:</P> <PRE>url: https://ransomwaretracker.abuse.ch/downloads/CW_C2_URLBL.txt ignore_regex: '^#' </PRE> <P>&nbsp;</P> <P>For a complete config example check <STRONG>dshield.block</STRONG> prototype.</P> <P><A target="_blank" name="user-content-minemeld-ft-csv-csvft"></A></P> <H1>&nbsp;</H1> <H2><A id="user-content-minemeldftcsvcsvft" class="anchor" href="https://github.com/PaloAltoNetworks/minemeld-core/blob/develop/docs/nodeconfig.rst#minemeldftcsvcsvft" target="_blank" rel="noopener">minemeld.ft.csv.CSVFT class</A></H2> <P>In addition to <STRONG>Base poller class</STRONG> config parameters, the CSVFT&nbsp;class supports the following parameters:</P> <P><A target="_blank" name="user-content-id4"></A></P> <H3>&nbsp;</H3> <H4><A id="user-content-parameters-2" class="anchor" href="https://github.com/PaloAltoNetworks/minemeld-core/blob/develop/docs/nodeconfig.rst#parameters-2" target="_blank" rel="noopener">Parameters</A></H4> <TABLE frame="void" rules="none"> <TBODY valign="top"> <TR> <TH>url:</TH> <TD>URL of the feed.</TD> </TR> <TR> <TH>polling_timeout:</TH> <TD>timeout of the polling request in seconds. Default: 20</TD> </TR> <TR> <TH>verify_cert:</TH> <TD>boolean, if <EM>true</EM> feed HTTPS server certificate is verified. Default: <EM>true</EM></TD> </TR> <TR> <TH>ignore_regex:</TH> <TD>Python regular expression for lines that should be ignored. Default: <EM>null</EM></TD> </TR> <TR> <TH>fieldnames:</TH> <TD>list of field names in the file. If <EM>null</EM> the values in the first row of the file are used as names. Default:<EM>null</EM></TD> </TR> <TR> <TH>delimiter:</TH> <TD>see <A href="https://docs.python.org/2/library/csv.html#dialects-and-formatting-parameters" target="_blank" rel="noopener">csv Python module</A>. Default: ,</TD> </TR> <TR> <TH>doublequote:</TH> <TD>see <A href="https://docs.python.org/2/library/csv.html#dialects-and-formatting-parameters" target="_blank" rel="noopener">csv Python module</A>. Default: true</TD> </TR> <TR> <TH>escapechar:</TH> <TD>see <A href="https://docs.python.org/2/library/csv.html#dialects-and-formatting-parameters" target="_blank" rel="noopener">csv Python module</A>. Default: null</TD> </TR> <TR> <TH>quotechar:</TH> <TD>see <A href="https://docs.python.org/2/library/csv.html#dialects-and-formatting-parameters" target="_blank" rel="noopener">csv Python module</A>. Default: "</TD> </TR> <TR> <TH>skipinitialspace:</TH> <TD>see <A href="https://docs.python.org/2/library/csv.html#dialects-and-formatting-parameters" target="_blank" rel="noopener">csv Python module</A>. Default: false</TD> </TR> </TBODY> </TABLE> <P><A target="_blank" name="user-content-id9"></A></P> <H3>&nbsp;</H3> <H4><A id="user-content-example-3" class="anchor" href="https://github.com/PaloAltoNetworks/minemeld-core/blob/develop/docs/nodeconfig.rst#example-3" target="_blank" rel="noopener">Example</A></H4> <P>Example config in YAML:</P> <PRE>url: https://sslbl.abuse.ch/blacklist/sslipblacklist.csv ignore_regex: '^#' fieldnames: - indicator - port - sslblabusech_type </PRE> <P>&nbsp;</P> <P>For a complete config example check <STRONG>sslabusech.ipblacklist</STRONG> prototype.</P> <P><A target="_blank" name="user-content-minemeld-ft-json-simplejson"></A></P> <H1>&nbsp;</H1> <H2><A id="user-content-minemeldftjsonsimplejson" class="anchor" href="https://github.com/PaloAltoNetworks/minemeld-core/blob/develop/docs/nodeconfig.rst#minemeldftjsonsimplejson" target="_blank" rel="noopener">minemeld.ft.json.SimpleJSON class</A></H2> <P>In addition to <STRONG>Base poller class</STRONG> config parameters, the SimpleJSON&nbsp;class support the following parameters:</P> <P><A target="_blank" name="user-content-id10"></A></P> <H3>&nbsp;</H3> <H4><A id="user-content-parameters-3" class="anchor" href="https://github.com/PaloAltoNetworks/minemeld-core/blob/develop/docs/nodeconfig.rst#parameters-3" target="_blank" rel="noopener">Parameters</A></H4> <TABLE frame="void" rules="none"> <TBODY valign="top"> <TR> <TH>url:</TH> <TD>URL of the feed.</TD> </TR> <TR> <TH>polling_timeout:</TH> <TD>timeout of the polling request in seconds. Default: 20</TD> </TR> <TR> <TH>verify_cert:</TH> <TD>boolean, if <EM>true</EM> feed HTTPS server certificate is verified. Default: <EM>true</EM></TD> </TR> <TR> <TH>extractor:</TH> <TD>JMESPath expression for extracting the indicators from the JSON document. Default: @</TD> </TR> <TR> <TH>indicator:</TH> <TD>the JSON attribute to use as indicator. Default: indicator</TD> </TR> <TR> <TH>fields:</TH> <TD>list of JSON attributes to include in the indicator value. If <EM>null</EM> no additional attributes are extracted. Default: <EM>null</EM></TD> </TR> <TR> <TH>prefix:</TH> <TD>prefix to add to field names. Default: json</TD> </TR> </TBODY> </TABLE> <P><A target="_blank" name="user-content-id11"></A></P> <H3>&nbsp;</H3> <H4><A id="user-content-example-4" class="anchor" href="https://github.com/PaloAltoNetworks/minemeld-core/blob/develop/docs/nodeconfig.rst#example-4" target="_blank" rel="noopener">Example</A></H4> <P>Example config in YAML:</P> <PRE>url: https://ip-ranges.amazonaws.com/ip-ranges.json extractor: "prefixes[?service=='AMAZON']" prefix: aws indicator: ip_prefix fields: - region - service </PRE> <P>&nbsp;</P> <P>For a complete config example check <STRONG>aws.AMAZON</STRONG> prototype.</P> Wed, 19 Feb 2025 11:20:48 GMT lmori 2025-02-19T11:20:48Z https://live.paloaltonetworks.com/t5/general-articles/configuring-nodes/ta-p/77185 <P>The set of config parameters supported by a node depends on the node class. Node configs are stored inside prototypes.</P> Wed, 19 Feb 2025 11:20:48 GMT https://live.paloaltonetworks.com/t5/general-articles/configuring-nodes/ta-p/77185 lmori 2025-02-19T11:20:48Z https://live.paloaltonetworks.com/t5/general-articles/configuring-nodes/tac-p/182746#M440 <P>Hi,</P> <P>&nbsp;</P> <P>When extraction JSON fields, is it possible to realize a conversion ?</P> <P>in my case, I have a field "last_contact" : <SPAN><SPAN class="objectBox objectBox-string">"2017-10-19T07:59:11Z" and I would like to set the aged_out based on this value like:</SPAN></SPAN></P> <PRE>age_out: default: myPrefix_last_contact+5d</PRE> <P>&nbsp;</P> <P><SPAN><SPAN class="objectBox objectBox-string">But to realize this, I understood that myPrefix_last_contact must be Timestamp in milliseconds.</SPAN></SPAN></P> <P>&nbsp;</P> <P><SPAN><SPAN class="objectBox objectBox-string">So how can I convert this "2017-10-19T07:59:11Z" ? in the JSON field declaration ? or in the age_out default ?</SPAN></SPAN></P> <P>&nbsp;</P> <P>&nbsp;</P> Thu, 19 Oct 2017 11:48:24 GMT https://live.paloaltonetworks.com/t5/general-articles/configuring-nodes/tac-p/182746#M440 GVN2022 2017-10-19T11:48:24Z https://live.paloaltonetworks.com/t5/general-articles/configuring-nodes/tac-p/182814#M441 <P><a href="https://live.paloaltonetworks.com/t5/user/viewprofilepage/user-id/37517">@GVN2022</a>&nbsp;Could you, please, share a anonymized JSON record example from your feed?</P> Thu, 19 Oct 2017 18:55:46 GMT https://live.paloaltonetworks.com/t5/general-articles/configuring-nodes/tac-p/182814#M441 xhoms 2017-10-19T18:55:46Z