[How to] Reverse Ingeniering (Download a Streaming video)

[elmarcia]

Hi epvp, the other day was trying to watch some series in a streaming platform, (those free that are full of ads and shitty pop ups), anyway, i tryed to watch the movie but was too slow so i ask myself, what about if i download this f*ckin video, no download button of course. So i need to figure out how it works.

Step 1: (getting some data)
I opened chrome dev tools with network tab open so i can see what was going on, full reload the page, had a hard time with those stupid pop ups and finally got to the video again. Looking at network tab i see something that was strange so i take a closer look (Image 1).

Image 1

 Spoiler

A file named init-a1-x3.mp4 was requested along with
init-v1-x3.mp4
fragment-1-a1-x3.m4s
fragment-1-v1-x3.m4s
fragment-2-v1-x3.m4s


Step 2: (making a guess with the data)
Making a guess v1 means video file and a1 means audio track,
and init should be mp4 headers and fragments are the video/audio pieces we need to collect in order to make our video file.

Step 3: (Proving our theory)
To prove our theory is easy just make an http request in chrome to
init-v1.mp4 and init-a1.mp4 and save to a file

Image 2

 Spoiler

We get what it looks like a video with no content but it has its length so our guess was right its only a header file.
Now we need to check if it is a valid header, we can do this combining our header file (init) with fragment-1 and check if it has any content.

So lets do it, first we download fragment-1 with chrome, then we open an hex editor (i use HXD works great) to copy the contents of fragment-1 to init.

Image 3 and 4

 Spoiler

Text in red is the contents of fragment-1 pasted in init. Of course they go in the end of the file if not we will corrupt first one and we don't want that.

Note that we paste fragment-1-v1 into init-v1 (because they are same format)

When we run our mp4 with VLC (video viewer), we got video yay!, but it is just a fragment of course, so we need to get the others...

Step 4: (the strategy)
-We know that init files are the headers of video and audio file
-We also know that fragments contains video if its a v1 fragment and audio if is a1 fragment.
-We even known that fragments are encoded like this:
fragment-{number}-{type}-x3.m4s

We have to automate the task to download our video and audio fragments and save them to our init file sequantially (if not we have a bunch of pieces in wrong order, we can't understand the movie this way).


Step 5: (the scripting)
I choose node for this project (easy https request with axios) you can choose whatever language best suits for you

The code

 Spoiler

const axios = require("axios");
const fs = require("fs");

axios.defaults.headers.get["Accept-Encoding"] = "gzip, deflate, br";

let maxFragments = 251; //trying going forward in video to the end fragment 251 breaks (error 404 not found)

const URL = "sorry but not giving endpoint don't want to get banned :)"
const header = (type)=>`init-${type}-x3.mp4`;
const fragment = (number,type) => `fragment-${number}-${type}-x3.m4s`;
const request = (data) => `${URL}/${data}`

const MediaType = {
	"video":"v1",
	"audio":"a1",
}

function getHeaders(){
let promises = [];
//save audio and video headers file
promises.push(axios.get(request(header(MediaType.video)),{responseType: 'arraybuffer'}).then(res => {
	const data = res.data;
	fs.appendFileSync("movie.mp4",res.data);
}));
promises.push(axios.get(request(header(MediaType.audio)),{responseType: 'arraybuffer'}).then(res => {
	const data = res.data;
	fs.appendFileSync("movie_audio.mp4",res.data);
}));
return Promise.all(promises);
}

getHeaders().then(()=>{
	getVideoFragment(1);
	getAudioFragment(1);
});
//recursive fragment save
//bad of this code we are saving our fragments in file sync, that means we wait until download ends
//and we wait until fragment is saved to disc to search for next fragment
//its slow, but saving bunch of different files and reassemblyng in one file seems more code than i want to
//write so i keep it slow but simple
function getVideoFragment(i){
  axios.get(request(fragment(i,MediaType.video)),{responseType: 'arraybuffer'}).then(res => {
	  fs.appendFileSync("movie.mp4",res.data);
	  console.log("Saving video fragment " + i);
	  if(i <= maxFragments){
		getVideoFragment(i+1);
	  } 
  }
  );
}
function getAudioFragment(i){
	return axios.get(request(fragment(i,MediaType.audio)),{responseType: 'arraybuffer'}).then(res =>{
		fs.appendFileSync("movie_audio.mp4",res.data);
		console.log("Saving audio fragment " + i);
		 if(i <= maxFragments){
		  getAudioFragment(i+1);
	  } 
	});
}

Final result:

 Spoiler

If you get this far, thank you for taking your time reading this guide, i enjoy doing it and also enjoy working in this mini project.

Please leave your questions and feedback if needed

[DarkOPM]

#moved

[MQTT.]

looks interesting!
maybe some explanations about your written code or comments?

[elmarcia]

Originally Posted by Nguyen.

looks interesting!
maybe some explanations about your written code or comments?

Ok, maybe i just need to change the way the last part was made, we can divide the scripting parts in small objectives to achieve a bigger one (obtaining full video and audio)

Objectives:
1) Download header file and fragments files
3) Append fragment file to header file in order.

So lets begin:

Download header file

Axios is a library that runs in node that lets us make http requests, axios sintax is very simple, it needs:
- an URL
- a method (POST / GET / PATCH ...)
- request Config --

My code seems a bit messy because of the use of functions that call functions, let me simplify it a bit

axios.get(request(header(MediaType.video)),{responseType: 'arraybuffer'})

is equal to:

axios.get("siteurl/init-v1-x3.mp4"),{responseType: 'arraybuffer'})

Response type is needed because i got lots of bugs when saving file if not present.

What i did here to prevent hard coding every url that i need to request something, i split functionality in a bunch of functions.

If not familiar with javascript es6, this code is similar

const header = (type)=>`init-${type}-x3.mp4`;

to something like this in another language

function getHeader(type){
return "init-"+type+"-x3.mp4"
}

Yes i should named it getHeaderNameByVideoType or sth like this but it is what it is lol.


Moving on those functions construct our request URL based on params that we provided

So taking that in mind

axios.get(codeThatConstructOurURLForVideoHeaderFile()),{responseType: 'arraybuffer'})
axios.get(codeThatConstructOurURLForAudioHeaderFile()),{responseType: 'arraybuffer'})

Seems a bit nice.

Now the tricky part of this, since we are making a request to a web server, we cannot be sure of how long it will take that request to be finished, so we need to work async.

Luckily for us, axios and many more libraries use promises.
If you are not familiar with promises i will try to explain what they are in a few words: Its a function that is called async (meaning its execution won't halt the code untils is finished) and calls a success function or a fail function.

You can find more about promises here:

Looking back at our code

axios.get(codeThatConstructOurURLForVideoHeaderFile()),{responseType: 'arraybuffer'}).then((response)=>{
//function that is called when a promise complete successfully
//here we save our file yay!
}).catch(e=>{
//function that is called when something go wrong, where e is the error
//here we retry or end 
})

Saving our file is supper easy but there is a catch it may be async or sync, how we decide which one to use?

Well we know that our files should be written sequentially like this:

VIDEO.mp4
--VIDEO-HEADER
--VIDEO-FRAGMENT-1
--VIDEO-FRAGMENT-2

AUDIO.mp4
--AUDIO-HEADER
--AUDIO-FRAGMENT-1
--AUDIO-FRAGMENT-2

Using async functions can mess up the order if not used correctly.

Example:

function saveFragmentAsync(fragmentData){
fs.saveFileAsync("video.mp4",fragmentData);
}
saveFragmentAsync("VIDEO-HEADER");
saveFragmentAsync("VIDEO-FRAGMENT-1");
saveFragmentAsync("VIDEO-FRAGMENT-2");
saveFragmentAsync("VIDEO-FRAGMENT-3");

Could result in a file looking like this:

video.mp4
--VIDEO-FRAGMENT-2
--VIDEO-HEADER
--VIDEO-FRAGMENT-1
--VIDEO-FRAGMENT-3

Wait what? how?, using async functions doesn't guarantee to be run in the order that was provided, we can in fact save header and fragments in different files and construct with them our final file, but that seems like a lot of work and don't want that (of course its not but is more code).


So we decided to save our files sync

fs.appendFileSync("movie.mp4",res.data); //halt until saved to disk

We wait for the first two files headers to be downloaded (audioheader,videoheader)

Using Promise.all(arrayOfPromises)

And then we proceed to download each fragment, wait until saved to the end of the corresponding file (audio or video).

And thats it work done.

//ask a webserver for video fragment i, and save it to disk
function getVideoFragment(i){
  axios.get("urlforvideofragment i"),{responseType: 'arraybuffer'}).then(res => {
//save fragment to end of header file sync
	  fs.appendFileSync("movie.mp4",res.data);
	  console.log("Saving video fragment " + i);
	  if(i <= maxFragments){
//call recursive if not finished yet
		getVideoFragment(i+1);
	  } 
  }
  );
}

[Baaaawz ヅ]

For what for a video hoster is it? I always used video downloader professional for google chrome as extension.