User reports via API?

[Dshackelford]

I have been able to create a new user report via the API, but have not found a way to run the report and retreive the generated pdf.  Is this completely beyond the bounds of the API?

The reason I am trying to do this is because we have two separate user bases that need monitoring and action: students and faculty.  Each has it's own path to resolution involving different areas (Student Development for students, and HR for faculty/staff).  I would really like to give them rights to generate their own reports for users, but they would share the same reporting area, and be able to see reports for the user base they are not responsible for.  This is primarily a problem with the staff in Student Develpment being able to run reports on other faculty and staff.

So, I want to use the API to generate the user reports dynamically and on demand.  I can create a portlet in our portal that gives access to reports for specific users (ie. the ones that have already shown up in blocked URL reports) so that the person responsible can handle disputes that arise.  If the list only includes the users that they should be able to view, the portlet can create the user activity report on the fly, but I need a way to retreive the actual report.

I guess the second suggestion would be ACLs for managing access to reports.

[mbenoit]

Hi,

I do not believe that you can invoke the 'export to pdf' function for a custom report via the XML API.  Why not just execute the report, collect the XML response and present it to the user in a web interface? Will that address what you are looking to accomplish?

Marc

[mbenoit · Accepted solution]

I believe you already have this worked out but here is an example of the xpath required to build a custom report and how to invoke it.

See this xpath example:

For configuration:

xpath=/config/shared/reports/entry[@name='testvsys']&element=<type><appstat><sortby>nbytes</sortby><group-by>category-of-name</group-by><aggregate-by><member>name</member></aggregate-by><values><member>nsess</member><member>nbytes</member></values></appstat></type><period>last-hour</period><topn>25</topn><topm>10</topm><query>(vsys eq vsys2)</query>

To invoke the report:

https://<YOUR-IP>/esp/restapi.esp?key=<YOUR-KEY>&type=report&reporttype=custom&reportname=testvsys

output:

<response status="success"><report logtype="appstat" start="2011/08/18 12:40:22" start-epoch="1313696422" end="2011/08/18 13:40:21" end-epoch="1313700021" generated-at="2011/08/18 13:40:22" generated-at-epoch="1313700022" axis="category-of-name"><entry><category-of-name>general-internet</category-of-name><name>web-browsing</name><nsess>184154</nsess><nbytes>2867152744</nbytes></entry><entry><category-of-name>general-internet</category-of-name><name>yahoo-toolbar</name><nsess>399061</nsess><nbytes>2744519666</nbytes></entry><entry><category-of-name>general-internet</category-of-name><name>flash</name><nsess>1981</nsess><nbytes>360290573</nbytes></entry><entry><category-of-name>general-internet</category-of-name><name>ping</name><nsess>702451</nsess><nbytes>74824382</nbytes></entry><entry><category-of-name>general-internet</category-of-name><name>rss</name><nsess>2208</nsess><nbytes>65049615</nbytes></entry><entry><category-of-name>general-internet</category-of-name><name>bittorrent</name><nsess>568</nsess><nbytes>63954077</nbytes></entry> ...

[Dshackelford]

Ha!  Yes, that is almost exactly what I am looking for.

I need to get an easy way to see the repeat counts for a particular recognizable site.  I can get a count for a destination IP, or resolved address but that is meaningless to the user, but not the actual hostname from the URL.  Yet, the user activity report includes this, which is why I was trying to find a way to run and retrieve it.

[mbenoit]

Can you share with me what language you are scripting in and a sample of the data output that you are looking for from a UAR?  That will help me to identify if there is another way to get that data and present it in a usable way.  We are also currently working on a Web Services SDK that may help you.  Here is a snippet of a widget that shows unique hits for URL filtering report using the XML API.

[Dshackelford]

I am using PoSh at the moment, but portlets will probably be written in .Net.

I am looking to have something like the following:

Website	Repeat Count
www.playb0y.com	450
www.otherxxxsite.com	234
www.foreignxxxsite.fr	43
www.somexxxstreamingsite.com	15

These are the actual host names from the URLs, rather than the resolved host names from an IP.  We want to know what site the user was actually visiting, not just which IP, or shared host.

I am pretty sure I can do the work on my end by pulling a report of repeat count per destination IP, and then grabbing and matching each IP to a URL in order to pull the host name.  But getting the URLs associated with an IP might be the hard part if it is a separate report.

I am also sending all URL filtering logs into Splunk, and can use Splunk's API to pull the data realtime with all the details already indexed and ready to be formated, but I would rather not depend on an external tool if I don't have to.

By the way, the image of the widget in your post does not show up for me.

[mbenoit]

Hi,

I updated and re-linked the image in my other post so that you can see it.  The SDK is using PHP and EXTJS to create widgets for data presentation, although I do not see any challenges with your power shell/.NET approach.  From my perspective I would create a custom report against the URL filtering database and group by user, here is an example output:

Source User	Source address	Category	URL	Repeat Count
pancademo\patricia.enriqu	10.154.9.201	shopping	www.gmarket.co.kr/challenge/neo_include/dummy.asp	917
pancademo\patricia.enriqu	10.154.9.201	shopping	gen.gmarket.co.kr/challenge/neo_include/dummy.asp	768
pancademo\patricia.enriqu	10.154.9.201	shopping	ad.gmarket.co.kr/js.kti/G_etc/head@155_30	694
pancademo\patricia.enriqu	10.154.9.201	shopping	ad.gmarket.co.kr/html.kti/G_etc/head@155_30	624
pancademo\patricia.enriqu	10.154.9.201	shopping	gen.gmarket.co.kr/challenge/neo_goods/inventory_add_info_opt.asp	312
pancademo\patricia.enriqu	10.154.9.201	shopping	ad.gmarket.co.kr/js.kti/gmarket/bestseller@pv	263
pancademo\patricia.enriqu	10.154.9.201	shopping	image.gmarket.co.kr/challenge/neo_etc_main/best_seller/20090408/best_title_tab22_over.gif	216
pancademo\patricia.enriqu	10.154.9.201	shopping	gen.gmarket.co.kr/challenge/neo_goods/inventory_add_info.asp	192
pancademo\patricia.enriqu	10.154.9.201	shopping	image.gmarket.co.kr/challenge/neo_etc_main/best_seller/20090408/best_title_btn42_over.gif	192
pancademo\patricia.enriqu	10.154.9.201	shopping	ad.gmarket.co.kr/html.kti/gmarket/bestseller@pv	168
pancademo\james.young	10.154.14.6	business-and-economy	urs.microsoft.com/	719
pancademo\james.young	10.154.14.6	internet-portals	s.yimg.com/	408
pancademo\james.young	10.154.14.6	web-based-email	login.yahoo.com/	264
pancademo\james.young	10.154.14.6	internet-portals	www.aol.com/ads/load_v6.htm	216
pancademo\james.young	10.154.14.6	parked-domains	3rdmilclassrooms.com/CourseAWV2/flash.txt	192
pancademo\james.young	10.154.14.6	web-based-email	mail.google.com/a/horizon.yourcollege.edu/?ui=2&view=bsp&ver=1qygpcgurkovy	192
pancademo\james.young	10.154.14.6	search-engines	www.google.com/coop/cse/brand?form=cse-search-box&lang=en	168
pancademo\james.young	10.154.14.6	computer-and-internet-info	www.google-analytics.com/urchin.js	165
pancademo\james.young	10.154.14.6	parked-domains	3rdmilclassrooms.com/CourseAWV2/Images/bottom-shade.gif	144
pancademo\james.young	10.154.14.6	content-delivery-networks	a248.e.akamai.net/	144

That will allow you to extract data for all users of a specific type (you can add criteria to the report to isolate students from faculty) then you can take that output and carve it up appropriately and publish it to a secure website and ACL who has access to what in order to present the data to the appropriate users regarding violations/etc.

To create this custom report example using the XML API:

https://<YOUR-IP>/esp/restapi.esp?key=<YOUR-KEY>&type=config&action=set&xpath=/config/shared/reports&element=<entry name="test-user-gen-url-report"><type><url><sortby>repeatcnt</sortby><group-by>srcuser</group-by><aggregate-by><member>src</member><member>category</member><member>misc</member></aggregate-by><values><member>repeatcnt</member></values></url></type><period>last-hour</period><topn>10</topn><topm>10</topm></entry>

To invoke this example using the XML API:

https://<YOUR-IP>/esp/restapi.esp?key=<YOUR-KEY>&type=report&reporttype=custom&reportname=test-user-gen-url-report

Is this more in line with what you are looking to accomplish?

[Dshackelford]

The UAR includes a section called Browsing Summary by Website that produces exactly the data I want.  It does not count the times that a specific URL was accessed, but how many times a specific site was accessed.

The UAR seems to have generated fields from the URL log that are not available in the custom reports.

[mbenoit]

Hi,

I cannot seem to recreate that data field as it appears that the UAR is an amalgamation of several individual reports plus a few custom functions.  We could generate multiple API calls from different reports and then provide external logic to get the data you want.... 

Marc