Using MineMeld to filter YouTube videos to only approved playlists

by xhoms ‎07-06-2017 05:30 AM - edited ‎12-04-2017 09:39 AM (22,673 Views)


In some cases you might be asked to enable the YouTube application but only for a few set of videos or the ones in a given playlist. MineMeld includes an experimental miner prototype that can extract the video items in a YouTube playlist and convert them into a URL list that can be imported into your Internet Gateway Palo Alto Networks Firewall to achieve such a goal.


Theory of operations

There are three components that are needed to implement this use case:

  • A running MineMeld instance
  • A google account and a YouTube Data API v3 key (can be obtained for free from
  • An Internet Gateway PANOS device

The YouTube miner will use the provided API Key and the PlayList unique ID to grab the list of videos. They will be converted in a set of URL's ( that can be aggregated and placed into an output feed.

If you need additional videos then you can add a stdlib.listUrl miner node and manage your own exceptions.


The security policy in the FW will need two rules:

  • A bottom rule enabling google-base and youtube-base with a URL filtering profile blocking the url "". This way, although the streaming of videos is enabled, the YouTube application logic is broken at the time the user requests a given playback (/watch?v=...)
  • An upper rule overriding the URL filter for the list of videos provided by the MineMeld feed

A ssl decryption rule is needed to allow the FW full access to the URL details to successfuly apply the filters.


Step 1. Deploy MineMeld

First, visit and select the article (from the top right) about installing and running MineMeld appropriate to your environment. Note, if using the VMWare desktop instructions ( you can go ahead with the "Super fast setup" but please download the cloud-init ISO and mount it on first boot. Assuming an IP comes via DHCP and you have internet access, your VM will automatically be updated  to the latest version of Minemeld.


Make note of MineMelds IP address (from an ifconfig) and login from your browser (defaults to username: admin / password: minemeld)


Step 2. Get your YouTube Data API v3 key

Connect to and login with your Google Account. If this is the first time you access the cloud console then an empty dashboard will be shown.




Navigate to the API Manager -> Library and then click on YouTube Data API.





You can use any existing project to attach the YouTube API to. If this is your first time or if you do not have any project then you must create a new one. Each Google Account is provided with a quota of up to 12 projects for free.





After clicking the "Create" button you will be send back to the YouTube Data API screen where you must click on "Manage" to allow this project access YouTube data. Once enabled the "Create credential" button will be shown for you to move on.






Fill the form as shown in the following screen capture and click on "What credentials do I need?" to disclose the API key.





This is the API Key you'll be asked for in the step 4. So copy it to a safe place. 


Step 3. Get the ID of the YouTube playlist you want to mine.

This is the easiest part. We'll cover three common demands:

  • Videos from a known PlayList
  • Videos on a specific YouTube channel
  • Videos upload by a known User

In the case you already know, just navigate YouTube until you reach the playlist you want to enable and copy its ID from the URL.

The following screen captures are for the Ligthboard Series playlist in the Palo Alto Networks channel.







If you need to get the videos from a specific channel, then you can take advantage of an "under-the-hood" mapping in YouTube between the Channeld ID and its corresponding Upload Playlist ID. Let's take, as an example, the Live Community Channel in Palo Alto Networks' YouTube Account.




This channel's ID is UCPRouchFt58TZnjoI65aelA so its corresponding upload playlist id (containing the 87 videos uploaded to this channel) will be UUPRouchFt58TZnjoI65aelA. In other words: you just need to change the second character "C" with a "U".


Something equivalent happens for YouTube users. Each user in YouTube has an internal Channel ID that, if transformed, becomes the user's upload PlayList ID. To display any YouTube user internal Channel ID just play a video from that user and click on his name.




In this case we're using the Palo Alto Networks YouTube user as a example. Click on the user name to navigate to the YouTube user profile page. Note in the URL the Channel ID for this user.




In this case the Channel ID for the Palo Alto Networks user is UC2UPStk47kvhBn8P7Q5BaAg and its corresponding upload playlist ID (containing all videos uploaded by this user) will be UU2UPStk47kvhBn8P7Q5BaAg.


Step 4. Configure your MineMeld instance

The experimental YouTube paylist miner is provided as an external feature that can be added to your MineMeld instrance from


In this step you'll use the YouTube API Key and PlayList ID to configure the miner and generate the URL feed.

First click on "CONFIG" to expose your current configuration. In the bottom right part of the screen you'll locate the icon to access the prototype library. Open it and locate the "youtubeminer.playlistMiner" prototype.




Click on it and create a new prototype from that base




Fill the form using your YouTube API Key and Playlist ID from Steps 2 & 3.




The next step is to add the recently created prototype as an engine node. To achieve this just go back to the prototype library, locate the recently created one and click on "CLONE".






Provide a descriptive name for the node and click on "OK" to attach it to the engine's configuration.




Next step is attach a URL processor to the engine's configuration and to connect its input to the YouTube miner node. In this case we do not need to create a new prototype. Just "CLONE" the stdlib.aggregatorURL, provide a descriptive name for the node, and bind its input to the miner.




And, finally, we must attach an output node. We can just "CLONE" the stdlib.feedHCGreen and bind its input to the aggregator we deployed in the previous step.




Now it is time to commit the configuration and to check that the output node is publishing the expected list.









Step 5. Configure the PANOS Internet Gateway device

This is the last step and the documentation bellow is for a "green field" deployment and must be taken as guidance to modify your existing policy to provide this use case.


First of all a SSL decryption rule is needed to expose the URL details inside the YouTube application. In this example we're enabling "forward proxy" decryption for all SSL sessions from "trust" to "untrust" for the URL category "streaming-media" which the YouTube application belongs to. If this is yor first time with the SSL Decryption feature then look in the PANOS Knowledge Base for articles on how to configure it (Trust CA Certificates, Decryption Profiles, etc.)




You might want, as well, to deny the application named "quic". It maps to the experimental protocol used by Google's Chrome browser when accessing Google services like YouTube. Denying this application will force the browser to fail back to TLS and avoid Chrome user bypassing the decryption policy.


Next we need a "custom URL category" that targets the "" URL and a URL Filtering Profile that blocks it.fw2.png






We will use the URL Filtering profile in the first rule that enables "google-base" and also in the first rule that enables "youtube-base". If you do not have such a rules then just create a new one as shown in the following screen capture.




At this moment in time you might want to check that the YouTube application logic has been broken. Go to You should be able to navigate the application but a URL Filtering Block page will be shown as soon as you attempt to playback any given video.




Now we have to configure the PANOS device with an External Dynamic List connected to the MineMeld output feed created in the step 4.




And the corresponding URL Category must be used in a new security rule (above the previous one) to override the URL Filtering Profile for the videos in the playlist we want to enable. Note that we're using this URL Category as a matching criteria in the rule and not inside a URL Filtering Profile.




With this new configuration only the videos in the mined playlist should be enabled.







by scottsander
on ‎07-07-2017 05:26 AM

Can this work for YouTube channels as well as playlists?  I have a requirement to allow everyone in the organization access to a specific YouTube channel, not a playlist.

by xhoms
on ‎07-07-2017 10:34 AM

The current (experimental) class at only collects videos from playlists. But is looks quite easy to add a new class to use the Search:list method of the YouTube Data API v3 to collect videos from a channel. Let me know if you want to give it a try. Otherwise I'll add this task to my queue list.

by xhoms
on ‎07-12-2017 06:19 AM

@scottsander I've added a new class in the youtube-miner extension (version 0.4) to mine videos for a give channel-id. You can consume this class through the youtubeminer.ChannelMiner prototype.

by scottsander
on ‎07-18-2017 11:16 AM

@xhoms, thank you!

by davidaavilar
on ‎08-31-2017 11:34 AM

Hi, how often the miner refresh the feed? I have a channel added but I uploaded some more videos but they doesn't appear in the feed.

by xhoms
on ‎08-31-2017 11:49 AM

@davidaavilar all miners default to a 3600 polling interval. You can change that adding interval: <n> (in seconds)

 to the node configuration file.


There is also the option to force a manual pool using the small icon right to the "LAST RUN" status message that is shown when you click on the given miner from the Nodes tab in the WebUI

by davidaavilar
on ‎08-31-2017 12:44 PM

Thanks a lot!


by brucegarlock
on ‎09-02-2017 07:25 AM

Outstanding!  I have been needing to do this for a while, and now I can give this process to a department manager to curate approved work-related YouTube videos by sharing the Playlist with them.  This also gave me a chance to become familiar with MineMeld.  Articles like this are very helpful.

by praveen.manimozhi1
‎09-09-2017 05:16 AM - edited ‎09-09-2017 05:17 AM

@xhomsIf the number of videos is more than 500 am not getting all video from channel. example channel id 



by xhoms
on ‎09-14-2017 05:59 AM

@praveen.manimozhi1 : Just found a minute to check the channel id you pointed to and, although it has many items, as of today it only dumps the following 286 videos. Where did you get the +500 figure?


["-HjpL-Ns6_A", "nfs8NYg7yQM", "tt2k8PGm-TI", "nSDgHBxUbVQ", "QeaumjX9DNY", "k2qgadSvNyU", "AX8-YzMKZhQ", "nBmNcLBaPUE", "Ahha3Cqe_fk", "87gWaABqGYs", "nCkpzqqog4k", "Wv2rLZmbPMA", "1UQzJfsT2eo", "XjwZAa2EjKA", "1pfuHcIppnM", "VuNIsY6JdUw", "-kWHMH2kxXs", "TvNTcB6c-V8", "RqcjBLMaWCg", "1-8iAq848B8", "8IQbF0IK-f4", "0gWxHFMog9w", "IPfJnp1guPc", "K0ibBPhiaG0", "r2LpOUwca94", "rk_qLtk0m2c", "Fum3g86zUPc", "-FyjEnoIgTM", "LbKcHy9cav0", "ZKSWXzAnVe0", "Soa3gO7tL-c", "NziZ08Fw1wc", "v7MGUNV8MxU", "m0J-BwkQK4A", "XVMxZMxkAgM", "UeG1ftTmLAg", "by3yRdlQvzs", "_dK2tDK9grQ", "Cy0thWiV-eA", "fEOyePhElr4", "PMivT7MJ41M", "oJ09xdxzIJQ", "xWizsAgk_jI", "lwZqbQL4H4Q", "Vb_IRAuucxA", "wnJ6LuUFpMo", "g8S-Ee-TZzc", "lp-EO5I60KA", "2OFKM2G-dE8", "JGwWNGJdvx8", "Kb24RrHIbFk", "eNxO9MpQ2vA", "pB-5XG-DbAA", "76O3w4pt0CA", "ggR_SPwP6cw", "dGidYBqBHVw", "me5ihmdlAk4", "CTFtOOh47oo", "I9kUrTDReSU", "BpWoKTXbdvc", "AY9blLYMKnI", "-rey3m8SWQI", "2ihOXaU0I8o", "ztxEHtrgTRg", "-CmadmM5cOk", "EExwffrNBMg", "NZKXkD6EgBk", "YqeW9_5kURI", "_Ju6Q8Azcmg", "WsskVYisG60", "gm2_6DX_0Bw", "SFsHPFhtEYQ", "LZkWKqHNiYA", "dq37f5LUJgc", "KXsY2r1_9C0", "tsEoLVBk_Ts", "yZm8znUvLHA", "I9akzUc5_eY", "uuwfgXD8qV8", "VqEbCxg2bNI", "Dyg32hMf7Fk", "8xg3vE8Ie_E", "SOk-KVq-O1I", "p0DlTQ-bBMM", "ozv4q2ov3Mk", "oKu2FVy0oUo", "Ory6b2EJ3Bk", "Og5-Pm4HNlI", "_QdPW8JrYzQ", "iXexmZm3_j8", "FhlLC_PdnoQ", "tksqqmQ5_XY", "Um7pMggPnug", "F4Afusxc2SM", "iEZ6EqgIUkg", "fM8V1XOI-14", "H5_HVUS1l0Y", "NwCjSclxZIo", "7KHvKzpsQrA", "fyaI4-5849w", "kTHNpusq654", "7ecYoSvGO60", "PALMMqZLAQk", "k0BWlvnBmIE", "J0ldO87Pprc", "0eu3qpDFhjo", "3c4KhoCFxzA", "Zy6vBxqlapw", "qmNWjAeyvwU", "8nL4frtSLfM", "9Erdp5uyUMg", "ton42cZ6ZYU", "Cj3AV92fJ90", "Dlh-dzB2U4Y", "ikhQVNyZIRw", "xKJmEC5ieOk", "Ns167_llTiA", "bUT6S1xcd98", "_8ecy55NhMo", "WiTgn5QH_HU", "PEGccV-NOm8", "JZ07RGAHNh4", "q99rhZlYlAo", "fXhwm4p40wY", "vSW2M-BB1NE", "LgrXd0NM2y8", "2DBBb3_p8eg", "CD9qdF0KjTE", "-hdww59gQW0", "9u7RAxNZJvQ", "TiGbybsprjM", "mLZwB1kUXBI", "91ZSbnNy_jI", "HPCl5R__vk4", "u8B34VGC9-c", "cNi_HC839Wo", "wzH_WlN5xgM", "onRVRZCj0l4", "B3WVwy82UQQ", "2nAnT3PASak", "3a7reNuwFrU", "kJQP7kiw5Fk", "JtigV-biVGg", "JnR0iC0jFko", "eIVxPRqgtD4", "vOKZi2bKGFg", "3XW-XdDe6j0", "lGP1YFE5s4M", "jP1Yey2jOXM", "csOY21AUypE", "IxCxXU3YKWM", "cdg193GvnBA", "G4gQRWYAKG4", "dUwN6GI-0EQ", "oyEuk8j8imI", "JRnOiibGv9s", "GsF05B8TFWg", "QFH747sK200", "3tmd-ClpJxA", "CMdHDHEuOUE", "_RtxQupMP5U", "d14dRVW7RGY", "E7cl_t-0yuo", "onNahCXzONc", "Sq1z1P9AK78", "G_7JKVuC9Bw", "YY2yjEEoB3U", "t6ojAWT9rrE", "YLczIXECYZE", "qyObyQRsHe8", "bBty4F6nOI8", "iA7B06-3PM8", "QhErNFutfSc", "t006FTM9yXY", "ZLKifZnWLw8", "4TnQ99ESoKM", "UzzQlZZpBX4", "5JX665avjuY", "Wcnq-BgwXPw", "hX3j0sQ7ot8", "mfjGmBVAL-o", "f3YmD9NjbdI", "uNzdZbDUUoU", "YgFUNPyQjyI", "CQQcGkmuc3w", "-mQEJse-ngY", "qfT5r5jBPXc", "cneGA7HyCtU", "P_-xNFPo_5E", "KDmFbAhlh3w", "zPx5N6Lh3sw", "3zaivOMpFjg", "Fj8G9dGuNkU", "yIASosJbHaM", "au0B2545OG8", "K1_f-GqaHHo", "jCObZuaqWbM", "NuXaPB_KWg8", "2wNrh7rqIMw", "YZf4mpC4FHI", "iGk5fR-t5AU", "YF2WLsDPY7A", "-iSFM-T-F_0", "3h6gtgaUK6U", "F6s4l7eROxo", "GuM8vTq0jd4", "IETReChj4Xk", "QWnOF68HNMk", "ORGpup_xZZA", "oavayHUI68g", "f83OqPlao4c", "tdE2-xI3VNE", "dy8gw9MjhDo", "doPL3Z3ULw0", "i3djhnAiQtk", "-lX1QqbWB0Y", "MQ7zqhc3UQ4", "yC8pLBq9aRc", "5gFntKt8ABI", "uHiaFiHsEdM", "PRIwvlSVPmg", "J-WDjZvvyD4", "1zkYGvoqRKg", "aakSpSXLSYY", "D-Czd1q0I5k", "bJA7TH_VDwQ", "JplDNiSMHaM", "bXZEP6OwKBQ", "nQTwJEfd-5s", "ElbrFTwmFNA", "1Sm8A-u1vIA", "QCVxQ_3Ejkg", "uxBdyL0ZDiY", "jNQXAC9IVRw", "O9EuOxMMMqc", "vWPjL-YCClE", "Dqx9FYCfheI", "013BMMQAly0", "2wOz9MuwERs", "7QCYDzsQ_yM", "FYZIOLr-evE", "Zqwz14x9LwE", "byPUsTmY3JE", "xPNStcJYVsU", "ZN-1rqsWom4", "O8TMZQ3wnS4", "UFccvtrP1d8", "6B0hIJe1HbM", "E3sQS53QAK0", "5EvXrnxCseY", "To52aopvV_w", "HqupWn0rIQ8", "pg_sxSql1dk", "ha7ASaPnjbA", "ytitU9d_zhk", "eStrQ4NWQZs", "Rxy1zM5oBmI", "0FFEvjUXv2E", "_E8uCSi8zUc", "l5QNHLNCFjk", "2GngHQUaJbw", "0jUhJaTm7u8", "veU2Vds_TjA", "ONIDOJmunM4", "s-n1y_OScA8", "XGMj9-7gL_4", "e579xTT2YBc", "VFaxL9JCIjQ", "SrAxS9lLu20", "VHXIZhJaVdM", "-0DU_XaTGA0", "ECddFk0d7BI", "tos2uEbQw_0", "Xu0vQJHgwbk", "Q4JX1HX7kD0", "2G2cGav-dKI", "GS5V3J-Hovo", "r6T0M56TEMA", "A18GYy4OuE8", "MTTzsPlNPZs", "qWbz1gvOzJk", "i0znRxDm4Ks", "cyYn91ArUm0", "yGJN6I4AF40", "aemGqy0qjyo", "8m65z81kob8"]
by praveen.manimozhi1
on ‎09-14-2017 06:39 AM

@xhoms I rember getting 300 links for this channel, for exact number i have to check. The number 500 was the maximum that i got when i tried different channels. I also tried to mine the list of videos based on VideoCategoryId and the highest i managed to get was 522.

by xhoms
on ‎09-14-2017 09:46 AM

@praveen.manimozhi1 : I've digged a bit more into this. Looks like the YouTube Data API v3 (the one used by the miner to get the list of videos on a given channel) has limitations with searches that return more that 500 items. More info in


If you know of any way to get the items of a channel other that using the search method then I'll be more than happy to give it a try.

by xhoms
‎09-14-2017 11:04 PM - edited ‎09-15-2017 10:34 AM

@praveen.manimozhi1 : Found a way! For each channel or user there is an associated upload playlist that contains all the corresponding videos. You can leverage the playlistMiner to grab videos from these "upload" playlists.


Let's take the BBC News channel (UC16niRr50-MSBwiO3YDb3RA) as example. The first step is to use the YouTube DataV3 API to get the channel details.


 "kind": "youtube#channelListResponse",
 "etag": "\"VPWTmrH7dFmi4s1RqrK4tLejnRI/pu2EESTiSyWP2alpTaVy64kG8mc\"",
 "pageInfo": {
  "totalResults": 1,
  "resultsPerPage": 1
 "items": [
   "kind": "youtube#channel",
   "etag": "\"VPWTmrH7dFmi4s1RqrK4tLejnRI/6nN8wR9WB2xaciwAgGJee7jH9FY\"",
   "id": "UC16niRr50-MSBwiO3YDb3RA",
   "contentDetails": {
    "relatedPlaylists": {
     "uploads": "UU16niRr50-MSBwiO3YDb3RA",
     "watchHistory": "HL",
     "watchLater": "WL"

Using the youtube.playlistMiner I've been able to grab the +7000 videos from this channel.


The strategy also works for YouTube users. I've just successfuly mined +21000 videos from the BBC youtube user using its upload playlist.


 "kind": "youtube#channelListResponse",
 "etag": "\"VPWTmrH7dFmi4s1RqrK4tLejnRI/VyTUC9GxLFzJhwUDyZ7HVVD9R_U\"",
 "pageInfo": {
  "totalResults": 1,
  "resultsPerPage": 5
 "items": [
   "kind": "youtube#channel",
   "etag": "\"VPWTmrH7dFmi4s1RqrK4tLejnRI/icqs-U-GyQPE0aSG40SPGuZNSYE\"",
   "id": "UCCj956IF62FbT7Gouszaj9w",
   "contentDetails": {
    "relatedPlaylists": {
     "likes": "LLCj956IF62FbT7Gouszaj9w",
     "uploads": "UUCj956IF62FbT7Gouszaj9w",
     "watchHistory": "HL",
     "watchLater": "WL"


by praveen.manimozhi1
on ‎09-18-2017 03:02 AM

@xhoms I was able to mine 7000+ usrls with your idea. However when i try to mine video based on category. There is no luck, i used serach option to filter based on VideoCategoryId. Is there a way to filter videos based on Video Category ?


by xhoms
on ‎09-18-2017 06:56 AM

@praveen.manimozhi1 : from what I found in any strategy that relies on YouTube searches (like "search for all videos in a give category") will fail around 500 entries.


Looks like you're trying to achieve a very ambitious project: to limit YouTube surface by video category. Even if you found a way to get such a list, I guess the number of extries would be much larger than the number of entries a PANOS device can host in a custom category. So you should have to find a way to incorporate new videos as soon as the users demand them and age out after a given period of time. That could be achieved by a very specialized LocalDB sort of miner with the following workflow:

  • You implement a custom URL block page for the security rule that allows video in that given category. That custom captive portal would inform the user that the video that has just been blocked has been send to a queue for category verification.
  • You configure PANOS 8.0 the http log forwarding feature to trigger that specialized LocalDB miner to use YouTube Data API against the received URL to check the category. In case the category is the expected one then the video URL is added to the local DB with an expiration time with a TTL, for instance, of 1 month.
  • The next PANOS EDL URL poll will include all new videos that has been identified.
by danilo.souza
on ‎03-23-2018 08:11 AM



I have youtube-miner v4 installed in the Minemeld (virtual machine, not autofocus). I never had any problem until recently. I tried to mine an Youtube channel (ID: UCdMgFwmyOjhhWqf86BQtaBA) which has more than 30,000 videos, but my miner can get no more than 527. The status of the miner is 




Observing the log, I get for the last Indicator


TIMESTAMP 13/3/2018 10:15:47 -0300 #419240

"_age_out": 4294967295000,
"confidence": 100,
"share_level": "green",
"_last_run": 1520946881203,
"sources": [
"first_seen": 1520946881203,
"type": "URL",
"last_seen": 1520946881203


Some questions:


The youtube-miner v4 has any problem with channels with a large number of videos?

The POOLING indicator for LAST RUN is normal?

I would appreciate any help.


Thank you

by xhoms
on ‎03-23-2018 10:18 AM

Hi @danilo.souza,


the "channel" miner uses a search feature in the YouTube API that limits the amount of responses to around 500 entries. What you're experencing is a known issue.


But you can use the "playlist" miner instead. All "channels" in YouTube have a corresponding "upload" playlist you can use. Read the Section 3 of the article. In you case, the playlist ID is "UUdMgFwmyOjhhWqf86BQtaBA"

by danilo.souza
on ‎03-27-2018 05:27 AM

Hi xhoms,


thank you for the quick reply. That's right, it works the way you said. I thought that way the miner would mine videos listed and not listed in the channel, but it only get the listed ones. I'm not sure you know but many channels use youtube as repository, so the not listed uploaded videos are not able to be found by the youtube search engine.


Thank you very much.

by Peter_Sumner
on ‎04-03-2018 11:59 PM

I have downloaded mindmeld and find there is no youtube miner of any sort in the prototypes, I feel like a total goose now having invested half a day to now find it has been dropped from the available prototypes..  ver 0.9.44  is this now a dead end or is there a way to import a protoype?


Peter down under in Oz

by Pichit_Techawiset
on ‎04-04-2018 12:49 AM

Hi @Peter_Sumner,


Check the step 4. You have to manual download the YouTube miner from the Github by your minemeld.


the result after download and install should similar to attached picture

Screen Shot 2018-04-04 at 15.12.35.png

by Peter_Sumner
on ‎04-04-2018 07:58 PM

Thanks to 'pichit_techawiset' for pointing towards my missed step, I have worked my way through to the end of the instructions then found I wanted to change the setting for the playlist key I wanted to mine (I used the default in the example, sily me).  I could find no way to edit any of the miner details once save has been clicked, do you need to destroy the setup each time to make a small change like to the playlist key ?



by xhoms
on ‎04-04-2018 11:16 PM

Hi @Peter_Sumner,


current MineMeld WEB UI for the YouTube miner do not allow "hot changes". That means that, for miners of this class, any configuration change requires a brand new prototype to be created.


Many of us "feel your pain". This is the reason behind a MineMeld WEB UI reengineering project.

Ask Questions Get Answers Join the Live Community