Need help downloading stream only video from archive.org with youtube-dl

[Ilovetv9]

I am able to download some stream only videos from archive.org with youtube-dl with no problems. For example this downloads all the separate 1 minute .mp4 videos and combines them when finished - https://archive.org/details/WJLA_20010915_163000_ABC_Fall_Sneak_Peek_Special/

But another stream only that does not work is this video - https://archive.org/details/MSNBCW_20131125_040000_To_Catch_a_Predator/

The log says when trying to download "ERROR: unable to download video data: HTTP Error 403: Forbidden". It does not say that when downloading the first video link. What am I doing wrong? Why does the first stream only video link I can download with no problems, but the second link says its "Forbidden" How can I download? Is there another program that can download all stream only archive.org videos? Please and thanks for any help.

[Reino]

In the HTML-source in a "meta"-node you can find...
<meta property="og:video" content="https://archive.org/download/MSNBCW_20131125_040000_To_Catch_a_Predator/MSNBCW_20131125_040000_To_Catch_a_Predator.mp4">
...but strangely enough you'll get a "HTTP error 403 Forbidden" trying to open the video-url.

What about youtube-dl?
youtube-dl.exe -F "https://archive.org/details/MSNBCW_20131125_040000_To_Catch_a_Predator"
[archive.org] MSNBCW_20131125_040000_To_Catch_a_Predator: Downloading webpage
[archive.org] MSNBCW_20131125_040000_To_Catch_a_Predator: Downloading JSON metadata
[info] Available formats for MSNBCW_20131125_040000_To_Catch_a_Predator:
format code extension resolution note
0 mp4 640x360

youtube-dl.exe -g "https://archive.org/details/MSNBCW_20131125_040000_To_Catch_a_Predator"
https://archive.org/download/MSNBCW_20131125_040000_To_Catch_a_Predator/MSNBCW_20131125_040000_To_Catch_a_Predator.mp4?exact=1&start=0&end=120

The video on this website is only available as lots of artificially cut up segments.
youtube-dl scrapes the "embed"-variant (https://archive.org/embed/MSNBCW_20131125_040000_To_Catch_a_Predator/) of this website, which only has the first 2 minute segment of this video in the HTML-source.
The initial url you provided does have all the cut-up-segment urls in its HTML-source. These urls work, but you'll need a HTML/JSON parser to extract them.
Luckily xidel (http://videlibri.sourceforge.net/xidel.html) is a command-line tool that can do just that. I'm going to assume you're on Windows btw.

The value of the "value"-attribute in this "input"-node...
<input class="js-tv3-init" type="hidden" value='{...}'/>
...has all the information you want:
xidel.exe -s "https://archive.org/details/MSNBCW_20131125_040000_To_Catch_a_Predator" -e "//input[@class='js-tv3-init']/@value"
{"TV3.identifier":"MSNBCW_20131125_040000_To_Catch_a_Predator",[...]"TV3.aspectratio":1.7777777777778}
To parse this information as what it really is... JSON:
xidel.exe -s "https://archive.org/details/MSNBCW_20131125_040000_To_Catch_a_Predator" -e "parse-json(//input[@class='js-tv3-init']/@value)"
{
"TV3.identifier": "MSNBCW_20131125_040000_To_Catch_a_Predator",
"TV3.embedable": 0,
"TV3.ccnums": [".cc5", ".align", ".cc1", ".cc5"],
"TV3.ignore_me": 61,
"TV3.CLIP_SEC_MAX2": 60,
"TV3.CLIP_SEC_MAX3": 180,
"TV3.TVNRT": 0,
"TV3.thumbzillas": [
"000001",
[...]
"003646"
],
"TV3.clipstream_clips": [
"http://archive.org/download/MSNBCW_20131125_040000_To_Catch_a_Predator/MSNBCW_20131125_040000_To_Catch_a_Predator.mp4?t=0/60&ignore=x.mp4",
[...]
"http://archive.org/download/MSNBCW_20131125_040000_To_Catch_a_Predator/MSNBCW_20131125_040000_To_Catch_a_Predator.mp4?t=3600/3660&ignore=x.mp4"
],
"TV3.quotes": [],
"TV3.duration": "3660.66",
"TV3.aspectratio": 1.7777777777778
}

To extract all the urls from the "TV3.clipstream_clips"-array:
xidel.exe -s "https://archive.org/details/MSNBCW_20131125_040000_To_Catch_a_Predator" ^
-e "parse-json(//input[@class='js-tv3-init']/@value)/(TV3.clipstream_clips)()"
http://archive.org/download/MSNBCW_20131125_040000_To_Catch_a_Predator/MSNBCW_20131125_040000_To_Catch_a_Predator.mp4?t=0/60&ignore=x.mp4
[...]
http://archive.org/download/MSNBCW_20131125_040000_To_Catch_a_Predator/MSNBCW_20131125_040000_To_Catch_a_Predator.mp4?t=3600/3660&ignore=x.mp4

Use -f instead of -e to open/"follow" all the urls:
xidel.exe "https://archive.org/details/MSNBCW_20131125_040000_To_Catch_a_Predator" ^
-f "parse-json(//input[@class='js-tv3-init']/@value)/(TV3.clipstream_clips)()" ^
--download "{request-decode($url)/concat(extract(path,'.+/(.+)\.',1),'_',params/end div 60,'.mp4')}"

The --download expression translates to (which are downloaded to the current dir):
MSNBCW_20131125_040000_To_Catch_a_Predator_1.mp4
[...]
MSNBCW_20131125_040000_To_Catch_a_Predator_61.mp4

Alternatively you can do everything with a single extraction query:
xidel.exe -s --xquery ^"^
for $x at $i in parse-json(^
doc('https://archive.org/details/MSNBCW_20131125_040000_To_Catch_a_Predator')//input[@class='js-tv3-init']/@value^
)/(TV3.clipstream_clips)()^
return^
x:request({'url':$x})/file:write-binary(^
concat(extract(url,'.+/(.+)\.',1),'_',$i,'.mp4'),^
string-to-base64Binary(raw)^
)^
"

xidel is primarily not a file download tool, so there's no progress-bar. Use curl if you're desperate:
FOR /F "delims=" %A IN ('
xidel.exe -s "https://archive.org/details/MSNBCW_20131125_040000_To_Catch_a_Predator"
-e "parse-json(//input[@class='js-tv3-init']/@value)/(TV3.clipstream_clips)()"
') DO @curl.exe [options] "%A"

Once downloaded you can glue them all together with ffmpeg and its concat-demuxer. First create the concat list:
(FOR %A IN ("MSNBCW_20131125_040000_To_Catch_a_Predator_*.mp4") DO @ECHO file '%A') > mylist.txt
Then process with ffmpeg:
ffmpeg.exe -f concat -safe 0 -i "mylist.txt" -c copy "MSNBCW_20131125_040000_To_Catch_a_Predator.mp4"

BUT, there's a problem. At every 1 minute intersection you'll notice there's a small video fragment that's repeated.
As far as I can tell, there's no way to completely solve it, but I did find a way to make it less intrusive.

Sadly appending "?t=0/3661&ignore=x.mp4" doesn't work, but the extracted JSON does give away the maximum allowed segment length; the key "TV3.CLIP_SEC_MAX3" its value 180.
Appending "?t=0/180&ignore=x.mp4" does appear to work. I suggest you decrease that value by 1 second however for a smoother transition.
So with a segment length of 180 second (instead of 60) there will be 3 times less hiccups and by creating the start- and end numbers yourself the remaining transitions will be smoother.

xidel.exe -s "https://archive.org/details/MSNBCW_20131125_040000_To_Catch_a_Predator" ^
-e "parse-json(//input[@class='js-tv3-init']/@value)/(decimal(TV3.duration) div TV3.CLIP_SEC_MAX3)"
20.337

xidel.exe -s "https://archive.org/details/MSNBCW_20131125_040000_To_Catch_a_Predator" ^
--xquery ^"^
let $json:=parse-json(//input[@class='js-tv3-init']/@value),^
$amnt:=$json/(decimal(TV3.duration) div TV3.CLIP_SEC_MAX3)^
for $x in 0 to $amnt^
return^
join(^
(^
$x * 180,^
if ($x eq integer($amnt)) then^
ceiling($x * 180 + ($amnt mod integer($amnt) * 180))^
else^
($x + 1) * 180 - 1^
)^
)^
"
0 179
180 359
360 539
[...]
3420 3599
3600 3661

xidel.exe -s "https://archive.org/details/MSNBCW_20131125_040000_To_Catch_a_Predator" ^
--xquery ^"^
let $json:=parse-json(//input[@class='js-tv3-init']/@value),^
$amnt:=$json/(decimal(TV3.duration) div TV3.CLIP_SEC_MAX3)^
for $x in 0 to $amnt^
return^
concat(^
substring-before($json/(TV3.clipstream_clips)(1),'?'),^
'?t=',$x * 180,'/',^
if ($x eq integer($amnt)) then^
ceiling($x * 180 + ($amnt mod integer($amnt) * 180))^
else^
($x + 1) * 180 - 1,^
'^&amp;ignore=x.mp4'^
)^
"
http://archive.org/download/MSNBCW_20131125_040000_To_Catch_a_Predator/MSNBCW_20131125_040000_To_Catch_a_Predator.mp4?t=0/179&ignore=x.mp4
http://archive.org/download/MSNBCW_20131125_040000_To_Catch_a_Predator/MSNBCW_20131125_040000_To_Catch_a_Predator.mp4?t=180/359&ignore=x.mp4
[...]
http://archive.org/download/MSNBCW_20131125_040000_To_Catch_a_Predator/MSNBCW_20131125_040000_To_Catch_a_Predator.mp4?t=3420/3599&ignore=x.mp4
http://archive.org/download/MSNBCW_20131125_040000_To_Catch_a_Predator/MSNBCW_20131125_040000_To_Catch_a_Predator.mp4?t=3600/3661&ignore=x.mp4

xidel.exe "https://archive.org/details/MSNBCW_20131125_040000_To_Catch_a_Predator" ^
--follow-kind=xquery3 -f ^"^
let $json:=parse-json(//input[@class='js-tv3-init']/@value),^
$amnt:=$json/(decimal(TV3.duration) div TV3.CLIP_SEC_MAX3)^
for $x in 0 to $amnt^
return^
concat(^
substring-before($json/(TV3.clipstream_clips)(1),'?'),^
'?t=',$x * 180,'/',^
if ($x eq integer($amnt)) then^
ceiling($x * 180 + ($amnt mod integer($amnt) * 180))^
else^
($x + 1) * 180 - 1,^
'^&amp;ignore=x.mp4'^
)^
" ^
--download ^"{^
request-decode($url)/concat(^
extract(path,'.+/(.+)\.',1),^
'_',^
ceiling(params/end div 180),^
'.mp4'^
)^
}"

[videoh]

We can't help you to violate terms of service.

[manolito]

We can't help you to violate terms of service.

Which "terms of service" are you referring to? AFAIK the only thing whch matters here is a violation of the forum rules.

I believe that for archived TV previews the same guidelines apply as for YouTube downloads. If you can watch it legally (which I assume is true for archived TV previews) then you can also record it legally to your VHS recorder, or todays equivalent which is download it to your HDD. Of course not for commercial purposes.

You may want to reread these old posts:
https://forum.doom9.org/showthread.php?p=1658338#post1658338

And another interesting post:
https://forum.doom9.org/showthread.php?p=1432768#post1432768
Normal users should just report posts or simply leave it to the mods This does not include former mods...

FWIW I had no problems downloading the clips from both linked URLs in the first post with IDM (Internet Download Manager). They have a 30 day trial period.

[Reino]

I had no problems downloading the clips from both linked URLs in the first post with IDM (Internet Download Manager).
Interesting. Did this IDM show you the url (or urls?) it was using?

[manolito]

Yes it does.

Clicking "Properties" for any downloaded clip IDM reveals the download URL. For one of the clips from the second link in the first post the URL which IDM reported was
https://ia800901.us.archive.org/19/items/MSNBCW_20131125_040000_To_Catch_a_Predator/MSNBCW_20131125_040000_To_Catch_a_Predator.mp4?start=300&end=360&ignore=x.mp4

[stax76]

The 'Joe Arpaio' of the Doom9 forum doing his thing.

[Reino]

So it's using the same urls* as I extracted from the "TV3.clipstream_clips"-array.
I don't know if you've downloaded some or all of them, but if you did, then you should notice the hiccup after every minute.

*xidel automatically follows a redirected url, so there's no need to do that beforehand:
xidel.exe -s --method=HEAD "http://archive.org/download/MSNBCW_20131125_040000_To_Catch_a_Predator/MSNBCW_20131125_040000_To_Catch_a_Predator.mp4?t=300/360&ignore=x.mp4" -e "$url"
http://ia600901.us.archive.org/19/items/MSNBCW_20131125_040000_To_Catch_a_Predator/MSNBCW_20131125_040000_To_Catch_a_Predator.mp4?start=300&end=360&ignore=x.mp4

curl.exe -Isw "%{redirect_url}\n" -o NUL "http://archive.org/download/MSNBCW_20131125_040000_To_Catch_a_Predator/MSNBCW_20131125_040000_To_Catch_a_Predator.mp4?t=300/360&ignore=x.mp4"
http://ia600901.us.archive.org/19/items/MSNBCW_20131125_040000_To_Catch_a_Predator/MSNBCW_20131125_040000_To_Catch_a_Predator.mp4?start=300&end=360&ignore=x.mp4

[manolito]

I don't know if you've downloaded some or all of them, but if you did, then you should notice the hiccup after every minute.

Yes, I also got these hiccups. But it was pretty straightforward to get rid of the repeated frames with AviDemux. Cutting only at I-Frames was all I needed for these clips.

[Reino]

I'm not familiar with AviDemux. Do you know if the same can be done with ffmpeg by any chance?

[manolito]

I believe ffmpeg uses a different approach than other tools when it comes to making cuts and edits. It uses timestamps to define the edit points instead of frame numbers. The ffmpeg GUIs I use (DMMediaConverter and WinFF) require to enter time points either in seconds or in the hh:mm:ss format, and there is no visual control for the user. The only ffmpeg based GUI I am aware of which lets you do this visually is DVDStyler. But there is no way to make multiple edits, only one start and end point can be defined.

AviDemux is quite similar to VirtualDub when it comes to editing. Multiple edit points are supported, full visual control, unlimited redo option. It is not frame accurate, though. For frame accurate editing without reencoding I prefer SmartCutter.

If you need to use AviDemux under WinXP, avidemux_2.6.8_win32_v2.exe works fine. Get it at VideoHelp. A slightly newer no-install version linked to by Mr. mean himself is here:
http://fixounet.free.fr/avidemux/avidemux_r020815_22c4_win32_winxp.7z
This is version 2.6.10. It does not come with AVSProxy.exe, if you need the proxy then you can extract the archive over an installed version 2.6.8.

[Ilovetv9]

Hey thanks Reino for looking at this.

I forgot about making this post, but today I tried to download the stream only predator videos from archive.org using tartube + yt-dlp because I read it is better than normal youtube-dl to download videos from the internet. But it still fails.

I don't understand why the youtube-dl people won't fix this problem. If the video is on archive.org with 1 minute mp4 files, I don't understand how it violates terms of service like videoh says to download the mp4 files and stich them together like youtube-dl does for other archive.org stream only videos that I showed in my original post.

Isn't that the whole point of youtube-dl - to download anything off the internet? So why is it so wrong to download a stream only video from archive.org where you can view the 1 minute videos anyway?

[videoh]

It's copyrighted material. You cannot discuss downloading that stuff per rule 6.