[stcogoli]

hi,
after reading the thread on rarred vobsubs, i thought why not implement unrarring more generally into the filesystem. so i've uploaded a patch as a proposal. from the readme:

----------------------------------
ok, say there exists some file "smb://ip/archive.rar" that contains a compressed file "folder\file.ext". you want to access "folder\file.ext" for reading operations.

1.)first step: create a rar url path using the function
  cutil::createrarpath(cstdstring& strurlpath, const cstdstring& strrarpath, const cstdstring& strfilepathinrar, const cstdstring& strpwd, const cstdstring& strcachepath = "z:\\"), where
 strrarpath is the path of the rar file("smb://ip/archive.rar"),
 strfilepathinrar is the relative path of the compressed file ("folder\file.ext"),
 strpwd is the archive password (strpwd = "" if the rar is not  encrypted)
strcachepath is a folder where temporary files will be copied (default is z:\
 strurlpath is a place holder that will be filled with the rar url, which is of the form
    strurlpath =  "rar://strcachepath,strrarpath,strfilepathinrar,strpwd"

2.) open a file supplying the above rar url:
cfile file;
file.open(strurlpath);
   the filesystem first finds the .rar. if it is on the network, it will be cached to strcachepath. then, it opens the .rar and extracts the file strfilepathinrar to the cache. all further file operations are redirected to this cached file so you can read it etc.. note however that compressing back into the original .rar is not supported so when you write to file, you are only modifying the cached version.

3.) when you are done, call
file.close()
   this will close the file and automatically delete the cached files.
-----------------------------------

i also got the following idea : when you open any file and the filesystem does not find it, look for a .rar of the same name which would contain that file and open it if it exists. that way, you could rar any file you want and the system would still open it.  i did not implement this cause i wasn't sure if it's  really a usefull/wanted feature
please post any suggestions/feedback.

[sergej]

great to see someone working on that.

i didn't look at your code too much or test it, but i just wanted to remind you that i didn't add any function to truncate the filename / remove illegal chars when extracting (stupid fatx). this should probably be done in extract.cpp, there already are functions called isnameusable and makenameusable, you just need to edit those.
i circumvent that problem since i first copy the rar to hd with a legal filename, and when the .sub is extracted it gets the same filename as the .rar (needed for vobsubs).
there already is a function in cutil to make the filename legal.

also, if you're extracting a file that contains ".sub" in the name (usually the ending), it will get extracted with the same name as the rar file, no matter what. we probably need a switch so this is only done if cachsubtitles is calling the unrar function.

and extracting directly from cd didn't work for me, had to copy to hd first. not sure about the network shares.

[stcogoli]

i just wanted to remind you that i didn't add any function to truncate the filename / remove illegal chars when extracting (stupid fatx)

ok. will add that.

also, if you're extracting a file that contains ".sub" in the name (usually the ending), it will get extracted with the same name as the rar file, no matter what. we probably need a switch so this is only done if cachsubtitles is calling the unrar function.

thanks for the tip, i wasn't aware of that. isn't it weird that this is implemented in the unrarring library itself and not left to the program?

and extracting directly from cd didn't work for me, had to copy to hd first. not sure about the network shares.

i think it only extracts from the hdd so the way i set it up is cache the rar whenever it's not on hdd.

[nheinemans]

good to see what you're working on. i've been thinking about this for a long time but i don't have the coding skills to build it myself.

where do you plan to cache the files? i can imaging you want to do some housekeeping in case the xbox crashes before the cached file is removed.

[stcogoli]

where do you plan to cache the files? i can imaging you want to do some housekeeping in case the xbox crashes before the cached file is removed.

for now, you can specify a location where to cache each file as you open it (in my previous post, strcachepath points to this cache) but you do have to close each file before turning off the box if you don't want to unnecessarily clutter the hdd.
you could imagine a more sophisticated caching mechanism but it's not clear to me wether it would be worth implementing. (you can always clear the cache manually from time to time if it ever gets out of control). for now, i wanted to keep things simple and see what people want before complicating things.

[elupus]

how about rar file browsing.. any rar file should be considered a directory with this, (atleast over samba and local. ccx stream has different ideas)

so say you open the "dir" smb://computer/share/test1.rar, it should then list files that are included in that rar, in some manner.

ex. the files in that directory would automatically be given the paths rar://[smb://computer/share/test1.rar]/subdir/filex.ext

that would be a fully qualified path to a file in an archive. wich also could be stepped back from.
any file opened, imo shouldn't be extracted first.. they could be extracted on demand, only problem would be seeking, which would force you to extract the file up to that point you seeked to. and then return.

this way would also allow for nested rar's, since the rar file system will use our emulated file system to read the data from the archive so a path like would then be created,
rar://[rar://[smb://computer/share/file.rar]/file2.rar]/subdir/file.ext

don't take this stuff as a must at all, it's just some ideas i've had. any form of rar file reading would be pretty sweet.

elupus

[sergej]

isn't it weird that this is implemented in the unrarring library itself and not left to the program?

it took me quite a while to figure out where to change the output filename (as there is no option to specify the output filename, only the output directory), so i just changed it there. i was aware that this was not the optimal solution, but as it only affects .sub files it's not that big a deal. but i made some comments in the code in case someone would want to extend the rar function.

it would probably be better to add a new argument like issub (default value = false) to the unrar function, so if we're using the unrar function from the cachesubtitles function, we just set it to true. or maybe you know a better way.

and btw, the original thread about porting the unrar lib to xbox is here, it also states that unzip doesn't work (even though the unrar lib theoretically supports it).

[stcogoli]

how about rar file browsing..

good idea, seems like the next natural thing to do..

any file opened, imo shouldn't be extracted first.. they could be extracted on demand,

hmmm not sure about this. the rar file has to respect the file system interface to be useful so it has to cache the files in one of the file interface functions. what other place makes sense except open()? if you open the file, in principle it's to do something with it. i'm not sure i understand how you would implement this caching on demand.

it would probably be better to add a new argument like issub (default value = false) to the unrar function, so if we're using the unrar function from the cachesubtitles function, we just set it to true. or maybe you know a better way.

i think the output filename should be made specifiable from the program code. will look into it.

[elupus]

well, depends on the interface to the unraring rutines.
you really wouldn't need to cache anything.
as the file is opened, you just prepare so that you have checked the rar file so the requested file excists. when starting the file position is at zero. so as a call is made to read, you start extracting the file in smaller chunks, then when the requested amount of data has been extracted, return it cache the amount of data that went over whatever was requested. on a subsequent call to read start to fill up with the buffered data that was left of from last call and if that isn't enough start up the unraring process where it left of.

this stuff probably requires some knowledge of how how a rar file is built and how it is extracted, but it would make for an awesome system.

elupus

[jaga]

if this could be called from a python script, it'd be the foundation for a great update/packaging system. great work.

[stcogoli]

uploaded  a new patch with the following changes:
-added support for remote archives with illegal fatx names.
-modified the cachesubtitles() function to use the new filesystem. so on playback of movie.ext, it will look for movie.rar and extract movie.sub to z:\subtitles.sub.
-fixed a bug causing cached files to sometimes remain on the hdd.
-changed the rar url format so that getfilename(urlpath)returns the name of the archived file.

sergej, i removed the part where you changed the extracted file if it contained "sub" and rewrote the makeusablename() and is usablename() unrarxlib functions to mimic those of xbmc. copying the extracted file may now be done by calling file.cache(source, dest) in the program code.

[sergej]

i just tested your latest patch (23.11). well, at least the subtitle function that is.

unraring to z: seems to work fine, however there are a few major issues:

- the idx isn't copied to z: or is already deleted when the .sub gets extracted. this causes vobsubs not to work at all, remember you need the idx and the sub files in order for them to work.
uncompressed vobsubs don't work anymore either, it says in the log that both files (idx and sub) are cached successfully, but when i check z:, none of the files are present. maybe check your bugfix to clear cached subtitles...

- it doesn't work if the .sub file inside the rar has a different filename than the movie name. this is sometimes the case, since when you rip it off the dvd, vobsub gives you a file called vts_01_01.sub or something like that, so ppl just rar that up and rename the rar. that is why i had that "hack" to rename .sub file inside the rar to the .rar filename. directvobsub for windows works fine with those files, too.

- i changed the code a bit where it checks the alternate subtitle directory for files, did you reverse that? it only makes sense if it checks for fatx legal filenames, if you have a 50 char movie on cd and want a subtitle together with it, you have no chance putting that with the same name in the alternate subtitle directory.

[stcogoli]

the idx isn't copied to z: or is already deleted when the .sub gets extracted. this causes vobsubs not to work at all, remember you need the idx and the sub files in order for them to work.

oupps didn't know that, i've never used vobsubs. i was focusing more on putting unrar in filesystem than getting vobsubs to work. i just mimicked the code that caches uncompressed subs and assumed it was ok when i saw the .subs  being successfully cached.

it doesn't work if the .sub file inside the rar has a different filename than the movie name.

i'm currently fixing this.

i changed the code a bit where it checks the alternate subtitle directory for files, did you reverse that?

yes, i just copy/pasted the existing code for uncompressed vobsubs and changed the file paths into rar urls, and you're right that it should only check for legal filenames.

it seems i uploaded a bit too quickly, there are also few other issues. basically, if you did multiple calls, like file.open(somefile), followed by file.exists(someotherfile), then the second call would mess up my data members and file.close() would screw up. i also realized that i was trying too much to  cram unrarring into the file interface. there are operations which one should be  able to call that do not fit into this interface, like simply listing the files in an archive. so i'm currently rewriting it and will test it more thoroughly before uploading this time :lol:

[stcogoli]

new patch uploaded. ripped from readme:
-moved the core unrar functionality into a crarmanager class with which you can now list files in an archive. however, i have also kept the crarfile object and simply made it into a crarmanager wrapper so you can still unrar very simply  using only the file interface, and most of the management is done under the hood.

- fixed support for rarred vobsubs. on movie playback, it will scan for a .rar with the same name as the movie and extract all rarred files with a subtitle extension to z:\subtitle.* tested rarred subs with illegal file names over samba and it worked. see readme for more details.

[jmarshall]

looks good.

i don't wanna dump more work on you, but i presume that .rar files now work like directories? (ie you can open them from the filemanager?) if not, would a cdirectory based wrapper of the crarmanager be easy enough to implement (i presume so?)

cheers,
jonathan