Talk:Shadow Files

Multi-level shadow-files

<worf> aren't subtitle files supposed to be a sub-file of the actual video files?

<worf> rather than being on the same level as them

<hrm> you might have a point there worf

<pelican> Linking subtitles to just one video file is stupid; any number makes more sense, but I'm not terribly keen on it anyway

<hrm> worf, but that would make the regular files a kind of shadow-files as well... I assumed we would keep with the current structure of the regular files, and just add the abstraction-layer of the shadow-files

<hrm> ah yeah, what pelican said is also a point

<worf> thats something i cant comment on

<pelican> Normally if you implement this kind of abstraction layer, you make it universal

<pelican> It's how relational databases work

<worf> yeah

[...]

<pelican> So basically... yes, you can restrict shadow files to files which need them, but it's probably going to be less efficient and it's certainly more code

I did a little thinking on this, and I think worf's suggestion holds merit. Consider the following scenario: Two split-episode files are raw, and they have subs. Currently this would look like this:

http://xs68.xs.to/pics/06075/current-subs-split.png

With my current suggestion for shadow-files, it would be like the following picture:

http://xs68.xs.to/pics/06075/shadow-subs-split.png

With worf's suggestion it would be like this:

http://xs68.xs.to/pics/06075/worf-shadow-subs-split.png

And I have to say that worf's is the most intuitive of them. Basically every single file in the DB, would in and of themselves actually be shadow-files, having the ability to have sub-files and/or parent-files. Note that I use plural here - restricting parent- and sub-files to only one per file would be bad. RAWs would only be able to have one single subtitle file, while in actuality, there are lots of RAWs that have more than one version of a subtitle file, and also, subtitle files would only be able to be associated with one single RAW, while they usually work with more than one (not to mention the "separate subtitles for many episodes" case (see below)). Note also that if ordinary files were in fact shadow-files, this would make the extra abstraction layer found in this:

be unnecessary.

Worf's version does however raise the question about what to do with separate subtitles for many episodes, and I assume the result would be something like this:

http://xs68.xs.to/pics/06075/worf-shadow-subs-manysubsinone.png

As you can see here, the actual RAWs do not need shadow-files to encompass them, since in this case it's one raw for each episode.

Still, when it comes to split-episodes the "original" verision of the shadow-files is necessary - you need the abstraction layer there.

I wasn't exactly sure where to put this discussion, but I figured this would be the best place for it. Maybe I should have posted to forum or something instead, but imo, the wiki works better for this.

--Suppy 03:17, 17 February 2006 (CET)

Also another case which would be relatively weird if you didn't have multi-layered support: .sub/.idx combinations for split episode files. Let me illustrate: Currently:

http://xs68.xs.to/pics/06076/current-subs-split-subandidx.png

With original shadow-files:

http://xs68.xs.to/pics/06076/shadow-subs-split-subandidx.png

With worf's shadow-files:

http://xs68.xs.to/pics/06076/worf-shadow-subs-split-subandidx.png

As you can see, this would require that every file be able to have files under it and be assigned to however many other files is necessary.

<pelican> A further consideration: subtitles must be available even if no other file for the episode or episodes they represent is

This would take care of this concern that pelican had. At least in my opinion. He did have a rather different approach to the problem than what is described here, and as I don't understand it fully, I can only hope that he will describe it properly himself.

---

Oh wait, I just realized that I didn't explain this part properly: "and be assigned to however many other files is necessary."
What I mean is that other RAWs for the same episode can usually use the same subs, so the files would have to be able to have many "parent" files, and the same subs should also be listed properly under the actual episode for those who just want the subs and don't need/already have/only want to steal/whatever can see that the episode actually has separate subs.

Hope this is actually helpful to some ppl, so I'm not just blowing off steam on a non-issue >_< ... I do feel that this needs a more proper going through, if exp agrees or not, I have no clue. It also seems unclear (at least to me) whether exp has decided on an actual design when it comes to shadow-files or if that is still up for discussion; there seems to be a few people who still has something to say about that, but at least PetriW did seem to describe the first of the shadow-files versions described here.

--Suppy 02:20, 18 February 2006 (CET)

Grouped shadow-files

pelican later came with a rather different suggestion...
paraphrased...

<pelican> The problem with subtitle association isn't the db structure (I could draw it in a minute if, um... if I could be bothered), it's the interface

<pelican> Things are already complicated by having shadow files which may represent multiple episodes

<pelican> A further consideration: subtitles must be available even if no other file for the episode or episodes they represent is

<pelican> They cannot be dependent

<pelican> Have fun.

<pelican> Oh, and even if exp implements shadow files (unlikely), subtitle association on top of that is probably more work than he's up for

<pelican> In case you're interested, the best structure is probably to have episode collections, which uniquely represent sets of episodes that shadow files use, put another layer on top of that representing same timing of an episode set and have subtitles act the same as normal files; you can group shadow files under one timing group with a mod change or (since this is an imaginary AniDB without the restrictions of the current) creq

<pelican> ...hope that line wasn't too long for the server

<pelican> All shadow files are then associated with a timing group

<pelican> The structure is (episode)>-<(episode set)-<(timed episode set)-<(shadow file)-<(file)

<pelican> Where >-< is many-many, -< one-many, and - would be one-one if I used it

<pelican> Current structure is (episode)-<(file)

<pelican> The episode sets are really just to allow us to quickly find timed episode sets of the same episodes

<pelican> This structure would require 6 tables, since many to many relationships need a relation table

[...]

<pelican> Ah, but shadow files group files together which are components of one shadow file; subtitles aren't components of another file, they're peripheral

<pelican> Having sub files under a/v is just wrong

<hrm> they're peripheral yes. but they're peripherals _to_ something. just like your mouse is a peripheral to your computer... you wouldn't go out and buy a mouse without either already having a computer or buying a computer at the same time.

<pelican> They're peripheral to the episodes

<pelican> Not to the files

<hrm> they are peripheral to the episodes, yes. but they are _also_ peripheral to the files.

<pelican> Read my suggested relations and you should see

<pelican> No, they are parallel to the files

<pelican> Files which represent the same group of episodes timed the same are interchangeable and can be used together where appropriate

<pelican> As with subtitles and a/v

<hrm> well, I didn't really get the whole thing with timed groups exactly. I would ask for more elaboration but I'm in the middle of watching a movie.

<pelican> It's just a set for which all (shadow) files would run in sync

<pelican> And thus in which you can apply sub files to a/v

<hrm> if you can explain to me how that would be done for something like this: http://xs68.xs.to/pics/06076/worf-shadow-subs-split-subandidx.png I'd appreciate it. that is supposed to be a 2cd release of a movie with accompanying .sub & .idx subtitle files.

<pelican> The .sub and .idx form a shadow file

<pelican> That shadow file is in the same timing group as raws whose timing it matches

<hrm> ah, I see. the way that would be handled in my system is that the .sub and .idx shadowfile would also be added to every other raw whose timing they match, as well as directly to the episode. .... it's just that for simplicity's sake, I only illustrated a single chain

<pelican> Best to only have one thing in one place

<pelican> And this automates association of them with other raws with the same timing

<hrm> yeah, that is a good point

In case you didn't exactly get all of that (it took a while before I did too ^^ (I'm still not sure I do ;D)), I'll do my best to illustrate it in a good way here ... Let's begin with the standard One file for One episode example, to get us started:

http://xs69.xs.to/pics/06082/groups-normal.png

Now... in case you don't understand why all the fuss is necessary here (I mean: what group - it's just one single file! just one single episode!) .. it's because of uniformity, so different code isn't needed for different cases ... besides, those layers would be all but invisible to the actual user. You'll see why they're actually needed in the next couple of examples.

Let's begin with the multi-episode file example:

http://xs69.xs.to/pics/06082/groups-multi.png

This should show you why the episode group is needed. Now let's move on to the timed group, which has more examples as to why it is needed.

Split-episode files:

http://xs69.xs.to/pics/06082/groups-split.png

I am not certain here whether the division should take place between (as I've illustrated) shadow-file and file, or if it should be between timed group and shadow-file. I'll haveto ask pelican to explain that more thoroughly. I believe I have guessed wrong here, but it was the way I did it, and it's too much of a bother to fix it now ;). If pelican says I'm wrong I'll do it however.

Moving on to the separate subs ...

We'll begin with the basic Separate subtitles for one episode example:

http://xs69.xs.to/pics/06082/groups-subs-normal.png

Here you can have any number of RAW files and subtitle files, as long as they're timed "for each other".

There is one problem that I haven't been able to figure out how to solve with this method, and that is: How to fix the "many subs in one file" problem. I asked pelican about it:
paraphrased (boldness added for focusing your eyes)

<hrm> hmm, with your episode sets and timed episode sets - how do you solve the "many subs in one file" problem illustrated here:

<hrm> I'm trying to illustrate the example of having a rar or zip file that contains many subs for more than one episode

[...]

<pelican> It doesn't deal with it; you can't expect shadow files to fix everything

<pelican> There is another structure that could work

<pelican> Move the timed structure to individual episodes

<pelican> However, this means that a single subtitles file which covers multiple episodes won't necessarily be timed correctly and it introduces further complications

<hrm> darn, that's what I feared you would say =/

<pelican> Your original idea of associating files and subtitles doesn't have this problem of course, but it's a fucking mess

<pelican> I wouldn't let it into any db I'd written

<pelican> (exp might, but he's never going to write any of this so that's not an issue)

"you can't expect shadow files to fix everything" - darn, I was hoping it would =/

This way of doing things doesn't solve everything, however it is a much cleaner way of fixing the shadow-files, and it solves most of the problems we currently have. A very viable option.

--Suppy 10:46, 21 February 2006 (CET)

This draft should contain all possible cases but one. Split files (Episode 0x Part 1 & Episode 0x Part 2) with both .sub and .idx subtitle files. We'd have to group the two split files under one shadow file but what to do with the four subtitle files? Grouping all four of them under one shadow file under the same timed group wouldn't be practical as we couldn't be sure which pair belongs to which split file.

--Worf 00:56, 22 February 2006 (CET)

Current DB Structure

Well, I thought it might benefit this topic to have a short description of the current DB structure. The relations look like this: (>-< N:M, -< 1:N, >- N:1)

Anime -< Episode -< File -< Stream

However, there are two additional relations as workarounds:

File >-< File To allow things like File-to-Subtitle relations, Files which form a unit or files which represent a newer version of another file.

Episode >-< File To allow files which span multiple episodes or files which span only part of an episode. The main relation remains Episode -< File mainly for performance reasons. Only a small subset of the files in the DB belong to more than one EP.

--Exp 03:56, 3 March 2006 (CET)