New YouTube seeds will have the default scoping rules automatically applied at the seed level when they are added to a collection. To learn more, including how you can add default scoping rules to existing seeds, please visit Sites with automated scoping rules.
You can add YouTube videos or channels to your collection in order to crawl, archive, and replay them as you would any other seed site, just so long as you remember to format and scope them according to a few simple rules:
On this page
How to format YouTube seeds
Watch pages
Specific videos on YouTube are hosted on a "watch" page with a URL in the following format: https://www.youtube.com/watch?v=XXXXXXXXXX. Follow this formatting and always use the "One Page" seed type to avoid scoping in all of youtube.com
Channels and Users
YouTube channels are topic-specific groups of videos and related content. For example, the University of Melbourne's channel can be accessed at https://www.youtube.com/user/unimelb/. This URL can serve as your seed URL. However, when you wish to archive a user's YouTube channel in its entirety, we further recommend adding an additional seed URL for its "Videos" tab, which is formatted as: https://www.youtube.com/user/[user-name]/videos. This enables our crawler to access all videos uploaded to the user's account.
The Standard seed type is best for crawling the videos linked off of a channel page, however a test crawl is strongly recommended when crawling a channel for the first time. Depending on the number of videos, these seeds can be very data heavy. Consider using the test crawl to determine how much data you should allot to these seeds then limit your crawl by adding a data limit at the seed level.
Playlists
Playlists are specific lists of videos curated by a user from among their account's and/or other videos on YouTube. The URL for each playlist can be added as a One-Page seed, and is typically formatted as: https://www.youtube.com/playlist?list=XXXXXXXXXXXXXXXXXX
To archive the playlist itself and the videos to which it links, it is necessary to add each linked video's Watch page (ex. https://www.youtube.com/watch?XXXXXXXX...) as a One-Page seed URL. These seeds can be set to private so they don't appear as access points in public collections.
If the playlist you want to capture has a large number of videos, you might consider using a browser extension that allows you to copy and paste (or otherwise export) links on a given page in bulk. We have had luck using the Chrome extension, Linkclump.
Note: These instructions will allow users to access video pages from the playlist's homepage but it does not guarantee that linked video pages from within Watch pages will be captured. The linked video pages on the Watch pages use slightly different URLs so if you would like them to also be captured they would also need to be added as individual seeds.
Search results
YouTube search pages like https://www.youtube.com/results?search_query="william+shakespeare" are best crawled using the One Page Plus External Links (One Page +) seed type. We strongly recommend a test crawl when crawling search pages. As with channels, you may need to add a seed level data limit in order to avoid crawling excessive additional videos.
Embedded videos
Videos that are embedded into other sites should archive successfully as long as the "general scoping rules" below have been applied to that site's crawl.
Default scoping rules for YouTube
You can set up your crawls to archive videos from YouTube watch pages, channel pages, or embedded videos in other sites, by adding a few simple scope modifications at either the Collection or Seed level.
Robots.txt Blocks
YouTube blocks important page styling content and some video files with robots exclusions. To make sure that you are able to capture the look and feel of a YouTube page and/or any video content, you will therefore need to add the rules listed below.
If you wish to archive YouTube videos linked or hosted by any site in the course of your crawl, you must first modify your collection's crawl scope to ignore robots.txt files from the following hosts, exactly as they appear here:
- youtube.com [note: no www.] - This is one of the hosts that serves YouTube video files.
- googlevideo.com [note: no www.] - This is the primary host that serves YouTube video files.
- ytimg.com - This host serves stylesheet and JavaScript content necessary for playback.
- OR -
If you prefer to archive only those videos and video pages from the specific YouTube seed that you added to your collection: Ignore robots.txt for each seed.
Crawler traps
Sometimes, YouTube crawls can run into crawler traps and archive invalid URLs with seemingly endless combinations of repeating directories. This is the most common crawler trap that affects the host www.youtube.com, as in this example:
https://www.youtube.com/channel/UCrHC0hXTvYewidE9Q5AM_8w/_/im/HTTP/www/www/HTTP/HTTP/www/HTTP/HTTP/www/www/channels
This issue can be addressed in the seed scope by adding the following repeating directories regular expression to Block URL if it matches the regular expression:
^.*?(/.+?/).*?\1.*$|^.*?/(.+?/)\2.*$
Optional scoping rules for YouTube
Scoping Googlevideo.com
Host level data limits apply individually to each subdomain of that host. Because googlevideo files are served from a large number of individual subdomains, adding one rule to limit the amount of Googlevideo content often isn't enough. For example, a 1GB data limit on the host googlevideo.com could result in 1GB from r5---sn-n4v7sn7s.googlevideo.com, 1GB from r3---sn-n4v7sn7s.googlevideo.com, and so on.
The easiest way to limit the amount of googlevideo content is to identify the seeds from which googlevideos are being captured, and add seed level data limits to them. The size of the data limit will vary depending on your seed, so be sure to run test crawls to make sure you're capturing what you need.
What to expect from archived YouTube videos
Archived YouTube content in Wayback currently replays most reliably in Chrome. If you do not see your videos replaying as expected, please check the link in Chrome before submitting a support ticket. This issue stems from browser limitations; we will update this documentation when partners can expect replay parity across browsers.
Individual videos archived in the manner described above should play back within the page. Any video that was captured from that page will also be accessible via the "Videos" link in the Wayback banner.
Pages with multiple embedded YouTube videos will only be able to load and replay the first video listed. All other captured videos will, however, still be accessible though the "Videos" link in the Wayback banner, and likewise from the tab listing videos archived in your publicly accessible collections.
Please note that the crawler will only access one version of a video during a crawl. If a video exists in multiple places within a seed site, it will only play back from the location it was discovered by the crawler initially.
Comments
0 comments
Please sign in to leave a comment.