MineMeld to Filter YouTube Videos

xhoms · ‎07-06-2017

Note: Palo Alto Networks made an end-of-life announcement about the MineMeld™ application in AutoFocus™ on August 1, 2021. Please read this article to learn about our recommended migration options.

Overview

In some cases you might be asked to enable the YouTube application but only for a few set of videos or the ones in a given playlist. MineMeld includes an experimental miner prototype that can extract the video items in a YouTube playlist and convert them into a URL list that can be imported into your Internet Gateway Palo Alto Networks Firewall to achieve such a goal.

Theory of operations

There are three components that are needed to implement this use case:

A running MineMeld instance
A google account and a YouTube Data API v3 key (can be obtained for free from console.cloud.google.com)
An Internet Gateway PANOS device

The YouTube miner will use the provided API Key and the PlayList unique ID to grab the list of videos. They will be converted in a set of URL's (https://www.youtube.com/watch?v=...) that can be aggregated and placed into an output feed.

If you need additional videos then you can add a stdlib.listUrl miner node and manage your own exceptions.

The security policy in the FW will need two rules:

A bottom rule enabling google-base and youtube-base with a URL filtering profile blocking the url "www.youtube.com/watch". This way, although the streaming of videos is enabled, the YouTube application logic is broken at the time the user requests a given playback (/watch?v=...)
An upper rule overriding the URL filter for the list of videos provided by the MineMeld feed

A ssl decryption rule is needed to allow the FW full access to the URL details to successfully apply the filters.

Step 1. Deploy MineMeld

First, visit https://live.paloaltonetworks.com/t5/MineMeld/ct-p/MineMeld and select the article (from the top right) about installing and running MineMeld appropriate to your environment. Note, if using the VMWare desktop instructions (https://live.paloaltonetworks.com/t5/MineMeld-Articles/Running-MineMeld-on-VMWare-desktop/ta-p/72038) you can go ahead with the "Super fast setup" but please download the cloud-init ISO and mount it on first boot. Assuming an IP comes via DHCP and you have internet access, your VM will automatically be updated to the latest version of Minemeld.

Make note of MineMelds IP address (from an ifconfig) and login from your browser (defaults to username: admin / password: minemeld)

Step 2. Get your YouTube Data API v3 key

Connect to https://console.cloud.google.com and login with your Google Account. If this is the first time you access the cloud console then an empty dashboard will be shown.

Navigate to the API Manager -> Library and then click on YouTube Data API.

You can use any existing project to attach the YouTube API also. If this is your first time or if you do not have any project then you must create a new one. Each Google Account is provided with a quota of up to 12 projects for free.

After clicking the "Create" button you will be send back to the YouTube Data API screen where you must click on "Manage" to allow this project access YouTube data. Once enabled the "Create credential" button will be shown for you to move on.

Fill the form as shown in the following screen capture and click on "What credentials do I need?" to disclose the API key.

This is the API Key you'll be asked for in the step 4. So copy it to a safe place.

Step 3. Get the ID of the YouTube playlist you want to mine.

This is the easiest part. We'll cover three common demands:

Videos from a known PlayList
Videos on a specific YouTube channel
Videos upload by a known User

In the case you already know, just navigate YouTube until you reach the playlist you want to enable and copy its ID from the URL.

The following screen captures are for the Ligthboard Series playlist in the Palo Alto Networks channel.

If you need to get the videos from a specific channel, then you can take advantage of an "under-the-hood" mapping in YouTube between the Channel ID and its corresponding Upload Playlist ID. Let's take, as an example, the Live Community Channel in Palo Alto Networks' YouTube Account.

This channel's ID is UCPRouchFt58TZnjoI65aelA so its corresponding upload playlist id (containing the 87 videos uploaded to this channel) will be UUPRouchFt58TZnjoI65aelA. In other words: you just need to change the second character "C" with a "U".

Something equivalent happens for YouTube users. Each user in YouTube has an internal Channel ID that, if transformed, becomes the user's upload PlayList ID. To display any YouTube user internal Channel ID just play a video from that user and click on his name.

In this case we're using the Palo Alto Networks YouTube user as a example. Click on the user name to navigate to the YouTube user profile page. Note in the URL the Channel ID for this user.

In this case the Channel ID for the Palo Alto Networks user is UC2UPStk47kvhBn8P7Q5BaAg and its corresponding upload playlist ID (containing all videos uploaded by this user) will be UU2UPStk47kvhBn8P7Q5BaAg.

Step 4. Configure your MineMeld instance

The experimental YouTube paylist miner is provided as an external feature that can be added to your MineMeld instrance from https://github.com/PaloAltoNetworks/youtube-miner

In this step you'll use the YouTube API Key and PlayList ID to configure the miner and generate the URL feed.

First click on "CONFIG" to expose your current configuration. In the bottom right part of the screen you'll locate the icon to access the prototype library. Open it and locate the "youtubeminer.playlistMiner" prototype.

Click on it and create a new prototype from that base

Fill the form using your YouTube API Key and Playlist ID from Steps 2 & 3.

The next step is to add the recently created prototype as an engine node. To achieve this just go back to the prototype library, locate the recently created one and click on "CLONE".

Provide a descriptive name for the node and click on "OK" to attach it to the engine's configuration.

Next step is attach a URL processor to the engine's configuration and to connect its input to the YouTube miner node. In this case we do not need to create a new prototype. Just "CLONE" the stdlib.aggregatorURL, provide a descriptive name for the node, and bind its input to the miner.

And, finally, we must attach an output node. We can just "CLONE" the stdlib.feedHCGreen and bind its input to the aggregator we deployed in the previous step.

Now it is time to commit the configuration and to check that the output node is publishing the expected list.

Step 5. Configure the PAN-OS Internet Gateway device

This is the last step and the documentation bellow is for a "green field" deployment and must be taken as guidance to modify your existing policy to provide this use case.

First of all a SSL decryption rule is needed to expose the URL details inside the YouTube application. In this example we're enabling "forward proxy" decryption for all SSL sessions from "trust" to "untrust" for the URL category "streaming-media" which the YouTube application belongs to. If this is your first time with the SSL Decryption feature then look in the PANOS Knowledge Base for articles on how to configure it (Trust CA Certificates, Decryption Profiles, etc.)

You might want, as well, to deny the application named "quic". It maps to the experimental protocol used by Google's Chrome browser when accessing Google services like YouTube. Denying this application will force the browser to fail back to TLS and avoid Chrome user bypassing the decryption policy.

Next we need a "custom URL category" that targets the "www.youtube.com/watch" URL and a URL Filtering Profile that blocks it.

We will use the URL Filtering profile in the first rule that enables "google-base" and also in the first rule that enables "youtube-base". If you do not have such a rules then just create a new one as shown in the following screen capture.

At this moment in time you might want to check that the YouTube application logic has been broken. Go to www.youtube.com. You should be able to navigate the application but a URL Filtering Block page will be shown as soon as you attempt to playback any given video.

Now we have to configure the PAN-OS device with an External Dynamic List connected to the MineMeld output feed created in the step 4.

And the corresponding URL Category must be used in a new security rule (above the previous one) to override the URL Filtering Profile for the videos in the playlist we want to enable. Note that we're using this URL Category as a matching criteria in the rule and not inside a URL Filtering Profile.

With this new configuration only the videos in the mined playlist should be enabled.

scottsander · ‎07-07-2017

Can this work for YouTube channels as well as playlists? I have a requirement to allow everyone in the organization access to a specific YouTube channel, not a playlist.

xhoms · ‎07-07-2017

The current (experimental) class at https://github.com/PaloAltoNetworks/youtube-miner only collects videos from playlists. But is looks quite easy to add a new class to use the Search:list method of the YouTube Data API v3 to collect videos from a channel. Let me know if you want to give it a try. Otherwise I'll add this task to my queue list.

xhoms · ‎07-12-2017

@scottsander I've added a new class in the youtube-miner extension (version 0.4) to mine videos for a give channel-id. You can consume this class through the youtubeminer.ChannelMiner prototype.

scottsander · ‎07-18-2017

@xhoms, thank you!

davidaavilar · ‎08-31-2017

Hi, how often the miner refresh the feed? I have a channel added but I uploaded some more videos but they doesn't appear in the feed.

xhoms · ‎08-31-2017

@davidaavilar all miners default to a 3600 polling interval. You can change that adding interval: <n> (in seconds)

to the node configuration file.

There is also the option to force a manual pool using the small icon right to the "LAST RUN" status message that is shown when you click on the given miner from the Nodes tab in the WebUI

davidaavilar · ‎08-31-2017

Thanks a lot!

brucegarlock · ‎09-02-2017

Outstanding! I have been needing to do this for a while, and now I can give this process to a department manager to curate approved work-related YouTube videos by sharing the Playlist with them. This also gave me a chance to become familiar with MineMeld. Articles like this are very helpful.

praveen.manimozhi1 · ‎09-09-2017

@xhomsIf the number of videos is more than 500 am not getting all video from channel. example channel id

UCVTyTA7-g9nopHeHbeuvpRA

xhoms · ‎09-14-2017

@praveen.manimozhi1 : Just found a minute to check the channel id you pointed to and, although it has many items, as of today it only dumps the following 286 videos. Where did you get the +500 figure?

["-HjpL-Ns6_A", "nfs8NYg7yQM", "tt2k8PGm-TI", "nSDgHBxUbVQ", "QeaumjX9DNY", "k2qgadSvNyU", "AX8-YzMKZhQ", "nBmNcLBaPUE", "Ahha3Cqe_fk", "87gWaABqGYs", "nCkpzqqog4k", "Wv2rLZmbPMA", "1UQzJfsT2eo", "XjwZAa2EjKA", "1pfuHcIppnM", "VuNIsY6JdUw", "-kWHMH2kxXs", "TvNTcB6c-V8", "RqcjBLMaWCg", "1-8iAq848B8", "8IQbF0IK-f4", "0gWxHFMog9w", "IPfJnp1guPc", "K0ibBPhiaG0", "r2LpOUwca94", "rk_qLtk0m2c", "Fum3g86zUPc", "-FyjEnoIgTM", "LbKcHy9cav0", "ZKSWXzAnVe0", "Soa3gO7tL-c", "NziZ08Fw1wc", "v7MGUNV8MxU", "m0J-BwkQK4A", "XVMxZMxkAgM", "UeG1ftTmLAg", "by3yRdlQvzs", "_dK2tDK9grQ", "Cy0thWiV-eA", "fEOyePhElr4", "PMivT7MJ41M", "oJ09xdxzIJQ", "xWizsAgk_jI", "lwZqbQL4H4Q", "Vb_IRAuucxA", "wnJ6LuUFpMo", "g8S-Ee-TZzc", "lp-EO5I60KA", "2OFKM2G-dE8", "JGwWNGJdvx8", "Kb24RrHIbFk", "eNxO9MpQ2vA", "pB-5XG-DbAA", "76O3w4pt0CA", "ggR_SPwP6cw", "dGidYBqBHVw", "me5ihmdlAk4", "CTFtOOh47oo", "I9kUrTDReSU", "BpWoKTXbdvc", "AY9blLYMKnI", "-rey3m8SWQI", "2ihOXaU0I8o", "ztxEHtrgTRg", "-CmadmM5cOk", "EExwffrNBMg", "NZKXkD6EgBk", "YqeW9_5kURI", "_Ju6Q8Azcmg", "WsskVYisG60", "gm2_6DX_0Bw", "SFsHPFhtEYQ", "LZkWKqHNiYA", "dq37f5LUJgc", "KXsY2r1_9C0", "tsEoLVBk_Ts", "yZm8znUvLHA", "I9akzUc5_eY", "uuwfgXD8qV8", "VqEbCxg2bNI", "Dyg32hMf7Fk", "8xg3vE8Ie_E", "SOk-KVq-O1I", "p0DlTQ-bBMM", "ozv4q2ov3Mk", "oKu2FVy0oUo", "Ory6b2EJ3Bk", "Og5-Pm4HNlI", "_QdPW8JrYzQ", "iXexmZm3_j8", "FhlLC_PdnoQ", "tksqqmQ5_XY", "Um7pMggPnug", "F4Afusxc2SM", "iEZ6EqgIUkg", "fM8V1XOI-14", "H5_HVUS1l0Y", "NwCjSclxZIo", "7KHvKzpsQrA", "fyaI4-5849w", "kTHNpusq654", "7ecYoSvGO60", "PALMMqZLAQk", "k0BWlvnBmIE", "J0ldO87Pprc", "0eu3qpDFhjo", "3c4KhoCFxzA", "Zy6vBxqlapw", "qmNWjAeyvwU", "8nL4frtSLfM", "9Erdp5uyUMg", "ton42cZ6ZYU", "Cj3AV92fJ90", "Dlh-dzB2U4Y", "ikhQVNyZIRw", "xKJmEC5ieOk", "Ns167_llTiA", "bUT6S1xcd98", "_8ecy55NhMo", "WiTgn5QH_HU", "PEGccV-NOm8", "JZ07RGAHNh4", "q99rhZlYlAo", "fXhwm4p40wY", "vSW2M-BB1NE", "LgrXd0NM2y8", "2DBBb3_p8eg", "CD9qdF0KjTE", "-hdww59gQW0", "9u7RAxNZJvQ", "TiGbybsprjM", "mLZwB1kUXBI", "91ZSbnNy_jI", "HPCl5R__vk4", "u8B34VGC9-c", "cNi_HC839Wo", "wzH_WlN5xgM", "onRVRZCj0l4", "B3WVwy82UQQ", "2nAnT3PASak", "3a7reNuwFrU", "kJQP7kiw5Fk", "JtigV-biVGg", "JnR0iC0jFko", "eIVxPRqgtD4", "vOKZi2bKGFg", "3XW-XdDe6j0", "lGP1YFE5s4M", "jP1Yey2jOXM", "csOY21AUypE", "IxCxXU3YKWM", "cdg193GvnBA", "G4gQRWYAKG4", "dUwN6GI-0EQ", "oyEuk8j8imI", "JRnOiibGv9s", "GsF05B8TFWg", "QFH747sK200", "3tmd-ClpJxA", "CMdHDHEuOUE", "_RtxQupMP5U", "d14dRVW7RGY", "E7cl_t-0yuo", "onNahCXzONc", "Sq1z1P9AK78", "G_7JKVuC9Bw", "YY2yjEEoB3U", "t6ojAWT9rrE", "YLczIXECYZE", "qyObyQRsHe8", "bBty4F6nOI8", "iA7B06-3PM8", "QhErNFutfSc", "t006FTM9yXY", "ZLKifZnWLw8", "4TnQ99ESoKM", "UzzQlZZpBX4", "5JX665avjuY", "Wcnq-BgwXPw", "hX3j0sQ7ot8", "mfjGmBVAL-o", "f3YmD9NjbdI", "uNzdZbDUUoU", "YgFUNPyQjyI", "CQQcGkmuc3w", "-mQEJse-ngY", "qfT5r5jBPXc", "cneGA7HyCtU", "P_-xNFPo_5E", "KDmFbAhlh3w", "zPx5N6Lh3sw", "3zaivOMpFjg", "Fj8G9dGuNkU", "yIASosJbHaM", "au0B2545OG8", "K1_f-GqaHHo", "jCObZuaqWbM", "NuXaPB_KWg8", "2wNrh7rqIMw", "YZf4mpC4FHI", "iGk5fR-t5AU", "YF2WLsDPY7A", "-iSFM-T-F_0", "3h6gtgaUK6U", "F6s4l7eROxo", "GuM8vTq0jd4", "IETReChj4Xk", "QWnOF68HNMk", "ORGpup_xZZA", "oavayHUI68g", "f83OqPlao4c", "tdE2-xI3VNE", "dy8gw9MjhDo", "doPL3Z3ULw0", "i3djhnAiQtk", "-lX1QqbWB0Y", "MQ7zqhc3UQ4", "yC8pLBq9aRc", "5gFntKt8ABI", "uHiaFiHsEdM", "PRIwvlSVPmg", "J-WDjZvvyD4", "1zkYGvoqRKg", "aakSpSXLSYY", "D-Czd1q0I5k", "bJA7TH_VDwQ", "JplDNiSMHaM", "bXZEP6OwKBQ", "nQTwJEfd-5s", "ElbrFTwmFNA", "1Sm8A-u1vIA", "QCVxQ_3Ejkg", "uxBdyL0ZDiY", "jNQXAC9IVRw", "O9EuOxMMMqc", "vWPjL-YCClE", "Dqx9FYCfheI", "013BMMQAly0", "2wOz9MuwERs", "7QCYDzsQ_yM", "FYZIOLr-evE", "Zqwz14x9LwE", "byPUsTmY3JE", "xPNStcJYVsU", "ZN-1rqsWom4", "O8TMZQ3wnS4", "UFccvtrP1d8", "6B0hIJe1HbM", "E3sQS53QAK0", "5EvXrnxCseY", "To52aopvV_w", "HqupWn0rIQ8", "pg_sxSql1dk", "ha7ASaPnjbA", "ytitU9d_zhk", "eStrQ4NWQZs", "Rxy1zM5oBmI", "0FFEvjUXv2E", "_E8uCSi8zUc", "l5QNHLNCFjk", "2GngHQUaJbw", "0jUhJaTm7u8", "veU2Vds_TjA", "ONIDOJmunM4", "s-n1y_OScA8", "XGMj9-7gL_4", "e579xTT2YBc", "VFaxL9JCIjQ", "SrAxS9lLu20", "VHXIZhJaVdM", "-0DU_XaTGA0", "ECddFk0d7BI", "tos2uEbQw_0", "Xu0vQJHgwbk", "Q4JX1HX7kD0", "2G2cGav-dKI", "GS5V3J-Hovo", "r6T0M56TEMA", "A18GYy4OuE8", "MTTzsPlNPZs", "qWbz1gvOzJk", "i0znRxDm4Ks", "cyYn91ArUm0", "yGJN6I4AF40", "aemGqy0qjyo", "8m65z81kob8"]

praveen.manimozhi1 · ‎09-14-2017

@xhoms I rember getting 300 links for this channel, for exact number i have to check. The number 500 was the maximum that i got when i tried different channels. I also tried to mine the list of videos based on VideoCategoryId and the highest i managed to get was 522.

xhoms · ‎09-14-2017

@praveen.manimozhi1 : I've digged a bit more into this. Looks like the YouTube Data API v3 (the one used by the miner to get the list of videos on a given channel) has limitations with searches that return more that 500 items. More info in https://stackoverflow.com/questions/25918405/youtube-api-v3-page-tokens

If you know of any way to get the items of a channel other that using the search method then I'll be more than happy to give it a try.

xhoms · ‎09-14-2017

@praveen.manimozhi1 : Found a way! For each channel or user there is an associated upload playlist that contains all the corresponding videos. You can leverage the playlistMiner to grab videos from these "upload" playlists.

Let's take the BBC News channel (UC16niRr50-MSBwiO3YDb3RA) as example. The first step is to use the YouTube DataV3 API to get the channel details.

https://www.googleapis.com/youtube/v3/channels?key=AIzaSyAfESPqWWlqM_.....jiw8WB0sQczFyEsA&id=UC16niRr50-MSBwiO3YDb3RA&part=contentDetails

{
 "kind": "youtube#channelListResponse",
 "etag": "\"VPWTmrH7dFmi4s1RqrK4tLejnRI/pu2EESTiSyWP2alpTaVy64kG8mc\"",
 "pageInfo": {
  "totalResults": 1,
  "resultsPerPage": 1
 },
 "items": [
  {
   "kind": "youtube#channel",
   "etag": "\"VPWTmrH7dFmi4s1RqrK4tLejnRI/6nN8wR9WB2xaciwAgGJee7jH9FY\"",
   "id": "UC16niRr50-MSBwiO3YDb3RA",
   "contentDetails": {
    "relatedPlaylists": {
     "uploads": "UU16niRr50-MSBwiO3YDb3RA",
     "watchHistory": "HL",
     "watchLater": "WL"
    }
   }
  }
 ]
}

Using the youtube.playlistMiner I've been able to grab the +7000 videos from this channel.

The strategy also works for YouTube users. I've just successfuly mined +21000 videos from the BBC youtube user using its upload playlist.

https://www.googleapis.com/youtube/v3/channels?key=AIzaSyAfESPqWWlqM_.....jiw8WB0sQczFyEsA&forUsername=BBC&part=contentDetails

{
 "kind": "youtube#channelListResponse",
 "etag": "\"VPWTmrH7dFmi4s1RqrK4tLejnRI/VyTUC9GxLFzJhwUDyZ7HVVD9R_U\"",
 "pageInfo": {
  "totalResults": 1,
  "resultsPerPage": 5
 },
 "items": [
  {
   "kind": "youtube#channel",
   "etag": "\"VPWTmrH7dFmi4s1RqrK4tLejnRI/icqs-U-GyQPE0aSG40SPGuZNSYE\"",
   "id": "UCCj956IF62FbT7Gouszaj9w",
   "contentDetails": {
    "relatedPlaylists": {
     "likes": "LLCj956IF62FbT7Gouszaj9w",
     "uploads": "UUCj956IF62FbT7Gouszaj9w",
     "watchHistory": "HL",
     "watchLater": "WL"
    }
   }
  }
 ]
}

praveen.manimozhi1 · ‎09-18-2017

@xhoms I was able to mine 7000+ usrls with your idea. However when i try to mine video based on category. There is no luck, i used serach option to filter based on VideoCategoryId. Is there a way to filter videos based on Video Category ?

xhoms · ‎09-18-2017

@praveen.manimozhi1 : from what I found in https://stackoverflow.com/questions/25918405/youtube-api-v3-page-tokens any strategy that relies on YouTube searches (like "search for all videos in a give category") will fail around 500 entries.

Looks like you're trying to achieve a very ambitious project: to limit YouTube surface by video category. Even if you found a way to get such a list, I guess the number of extries would be much larger than the number of entries a PANOS device can host in a custom category. So you should have to find a way to incorporate new videos as soon as the users demand them and age out after a given period of time. That could be achieved by a very specialized LocalDB sort of miner with the following workflow:

You implement a custom URL block page for the security rule that allows video in that given category. That custom captive portal would inform the user that the video that has just been blocked has been send to a queue for category verification.
You configure PANOS 8.0 the http log forwarding feature to trigger that specialized LocalDB miner to use YouTube Data API against the received URL to check the category. In case the category is the expected one then the video URL is added to the local DB with an expiration time with a TTL, for instance, of 1 month.
The next PANOS EDL URL poll will include all new videos that has been identified.

danilo.souza · ‎03-23-2018

Hi

I have youtube-miner v4 installed in the Minemeld (virtual machine, not autofocus). I never had any problem until recently. I tried to mine an Youtube channel (ID: UCdMgFwmyOjhhWqf86BQtaBA) which has more than 30,000 videos, but my miner can get no more than 527. The status of the miner is

STATUS
STATE STARTED
LAST RUN POLLING
# INDICATORS 527

Observing the log, I get for the last Indicator

TIMESTAMP 13/3/2018 10:15:47 -0300 #419240
TYPETRACE / EMIT_UPDATE
INDICATOR www.youtube.com/watch?v=0pBB0aLbqAU

VALUE
{
"_age_out": 4294967295000,
"confidence": 100,
"share_level": "green",
"_last_run": 1520946881203,
"sources": [
"youtube.channel"
],
"first_seen": 1520946881203,
"type": "URL",
"last_seen": 1520946881203
}

Some questions:

The youtube-miner v4 has any problem with channels with a large number of videos?

The POOLING indicator for LAST RUN is normal?

I would appreciate any help.

Thank you

xhoms · ‎03-23-2018

Hi @danilo.souza,

the "channel" miner uses a search feature in the YouTube API that limits the amount of responses to around 500 entries. What you're experencing is a known issue.

But you can use the "playlist" miner instead. All "channels" in YouTube have a corresponding "upload" playlist you can use. Read the Section 3 of the article. In you case, the playlist ID is "UUdMgFwmyOjhhWqf86BQtaBA"

danilo.souza · ‎03-27-2018

Hi xhoms,

thank you for the quick reply. That's right, it works the way you said. I thought that way the miner would mine videos listed and not listed in the channel, but it only get the listed ones. I'm not sure you know but many channels use youtube as repository, so the not listed uploaded videos are not able to be found by the youtube search engine.

Thank you very much.

Peter_Sumner · ‎04-03-2018

I have downloaded mindmeld and find there is no youtube miner of any sort in the prototypes, I feel like a total goose now having invested half a day to now find it has been dropped from the available prototypes.. ver 0.9.44 is this now a dead end or is there a way to import a protoype?

Peter down under in Oz

Pichit · ‎04-04-2018

Hi @Peter_Sumner,

Check the step 4. You have to manual download the YouTube miner from the Github by your minemeld.

the result after download and install should similar to attached picture

Screen Shot 2018-04-04 at 15.12.35.png

Peter_Sumner · ‎04-04-2018

Thanks to 'pichit_techawiset' for pointing towards my missed step, I have worked my way through to the end of the instructions then found I wanted to change the setting for the playlist key I wanted to mine (I used the default in the example, sily me). I could find no way to edit any of the miner details once save has been clicked, do you need to destroy the setup each time to make a small change like to the playlist key ?

Peter.

xhoms · ‎04-04-2018

Hi @Peter_Sumner,

current MineMeld WEB UI for the YouTube miner do not allow "hot changes". That means that, for miners of this class, any configuration change requires a brand new prototype to be created.

Many of us "feel your pain". This is the reason behind a MineMeld WEB UI reengineering project.

icartwright · ‎01-18-2019

So, I've got this all set up and running in my lab. I'm running into an issue where after browsing to Youtube and choosing a video, sometimes the PAN firewall identifies the app as "youtube-streaming". The video page shows, but the video itself doesn't play (because "youtube-streaming" isn't allowed in any rule, it gets blocked). This happens whether the video is supposed to be allowed or not (neither rule matches because neither of them have "youtube-streaming" in them).

If I add "youtube-streaming" to either rule, all Youtube videos are allowed when the application is "youtube-streaming" because that application doesn't get passed to the URL filtering engine (I think). There are no logs under Monitor > URL Filtering where the application is "youtube-streaming". There are some for both google-base and youtube-base.

What could be going on here?

I am running PANOS 8.1.5 on a VM-100 with current licenses for everything (if that helps).

Thanks.

upatino · ‎09-18-2019

Hello,

I am deploying this configuration but when I use it in Google Chrome, sometimes it blocks the video via "www.youtube.com/watch" and then if I refresh the page 2 or 3 times then it gets blocked.

I noticed in the logs that when it gets blocked the URL log is = www.youtube.com/watch?v=uQ3zA0oCFO4

When the video is not blocked, it gets= www.youtube.com/api/stats/playback?ns=yt&el=detailpage&cpn=P94yQGA71U_yFMjM&docid=uQ3zA0oCFO4&ver=2&...............

How can I get the videos blocked at first instance.

Thanks.

jdelio · ‎09-26-2019

This sounds like something that support will have to address, as I am unsure why that would be happening.

ovel · ‎12-24-2019

Hi xhomos,

First of all, thank you very much for this way of whitelisting youtube videos. It appears this is the only feasible way so far. I'm trying to implement some whitelists for children and while it works ok on a PC, there are problems with mobile devices:

1. In a mobile browser all videos appear to be starting from m.youtube.com, not www. So the dynamic list never matches. How can we get something like *.youtube.com instead of www?

2. While it seems to be working in browsers in PC mode on mobile devices we know that there is a special youtube app that people use. When I use ssl decryption, this app simply doesn't show anything. When I switch off the ssl decryption, the videos are not getting filtered. So the only working, and quite ugly way is to block the app from running and enforce watching videos in the browsers only. Is there any way to get the app and filtering working properly?

ovel · ‎12-29-2019

Hi Everyone,

I have fixed #1 from my previous question. I'm not good with github and don't have clear understanding of how it works, so I cloned the original repository in my own, updated the source python code a little bit and fed it to Minemeld. So the videos are being played successfully on both PC and mobile browsers (not youtube app tho). If you want, here is the souce to my repository on github: https://github.com/ovel1/youtube-miner

Re filtreing the native youtube app, so far the only way I found is integrating the childrens' mobile devices into Kaspersky Safe Kids environment and simply switching the app off. So that allows the kids to play only selected videos in browsers.

Edson-Teixeira · ‎05-21-2020

Good Morning !
I am experiencing a problem regarding quotas on Google Apis.

Is it possible to decrease the consultations so as not to exceed the limit?

ovel · ‎05-21-2020

There are actually 2 ways:

- update python code and then minemeld settings to reduce the number of mining attempts so that it will reduce the number of queries to youtube API.

- create multiple youtube accounts and use different API keys.

I did the latter. Effectively, youtube resets the counters every 24-hour cycle. The only thing you might miss is if someone during theis remaining 24-hours cycle has uploaded a new video into the channel you subscribed. These videos will be invisible till the next day.

JamesRen · ‎08-18-2020

Hi @xhoms, great article, thanks for sharing the use case.

Just one question here, sorry if I missed anything here but you have your URL filtering configured with 'allow' action for your URL Custom Category both both policies, I don't see why the second policy would block "www.youtube.com/watch". Should your URL filtering policy be set to 'block' instead of 'allow'?

Regards,

James

xhoms · ‎08-19-2020

Hi @JamesRen , I think you're right. The "/watch" policy is meant to deny non-allowed videos and so its action should be "block"

brucegarlock · ‎12-14-2021

Hello all,

With Minemeld EOL, is there another way to achieve this functionality?

hatus.cordeiro · ‎09-30-2022

Is it possible to do this for video categories?
I have youtube enabled, but I need to block all game channels!

Unlock your full community experience!

MineMeld to Filter YouTube Videos