Talk:Shadow Files: Difference between revisions

From AniDB
Jump to navigation Jump to search
(→‎Multi-level shadow-files: - how Tomynocker sees them)
mNo edit summary
 
(3 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{TOCright}}
==Multi-level shadow-files==
==Multi-level shadow-files==
<div style="border: 1px solid; background-color: #f2f2f2; padding: 5px;">
<div style="border: 1px solid; background-color: #f2f2f2; padding: 5px;">
<div style="border: 1px solid gray; background-color: #f9f9f9;"><tt>
<div style="border: 1px solid gray; background-color: #f9f9f9;"><tt>
:<worf> arent subtitle files supposed to be a sub-file of the actual video files?<br>
:<worf> aren't subtitle files supposed to be a sub-file of the actual video files?<br>
:<worf> rather than being on the same level as them<br>
:<worf> rather than being on the same level as them<br>
:<hrm> you might have a point there worf<br>
:<hrm> you might have a point there worf<br>
Line 22: Line 24:
With worf's suggestion it would be like this:
With worf's suggestion it would be like this:
<div style="width: auto; overflow: hidden;"><div style="border:1px solid #cccccc; padding: 3px; background-color:#f9f9f9; width: 482px;">http://xs68.xs.to/pics/06075/worf-shadow-subs-split.png</div></div>
<div style="width: auto; overflow: hidden;"><div style="border:1px solid #cccccc; padding: 3px; background-color:#f9f9f9; width: 482px;">http://xs68.xs.to/pics/06075/worf-shadow-subs-split.png</div></div>
And I haveto say that worf's is the most intuitive of them. Basically every single file in the DB, would in and of themselves actually be shadow-files, having the ability to have sub-files and/or parent-files. Note that I use plural here - restricting parent- and sub-files to only one per file would be bad. RAWs would only be able to have one single subtitle file, while in actuality, there are lots of RAWs that have more than one version of a subtitle file, and also, subtitle files would only be able to be associated with one single RAW, while they usually work with more than one (not to mention the "separate subtitles for many episodes" case (see below)). Note also that if ordinary files were in fact shadow-files, this would make the extra abstraction layer found in this:
And I have to say that worf's is the most intuitive of them. Basically every single file in the DB, would in and of themselves actually be shadow-files, having the ability to have sub-files and/or parent-files. Note that I use plural here - restricting parent- and sub-files to only one per file would be bad. RAWs would only be able to have one single subtitle file, while in actuality, there are lots of RAWs that have more than one version of a subtitle file, and also, subtitle files would only be able to be associated with one single RAW, while they usually work with more than one (not to mention the "separate subtitles for many episodes" case (see below)). Note also that if ordinary files were in fact shadow-files, this would make the extra abstraction layer found in this:
<div style="width: auto; overflow: hidden;"><div style="border:1px solid #cccccc; padding: 3px; background-color:#f9f9f9; width: 304px;">[[Image:Shadow-normal.png|none]]</div></div>
<div style="width: auto; overflow: hidden;"><div style="border:1px solid #cccccc; padding: 3px; background-color:#f9f9f9; width: 304px;">[[Image:Shadow-normal.png|none]]</div></div>
be unnecessary.
be unnecessary.
Line 28: Line 30:
Worf's version does however raise the question about what to do with separate subtitles for many episodes, and I assume the result would be something like this:
Worf's version does however raise the question about what to do with separate subtitles for many episodes, and I assume the result would be something like this:
<div style="width: auto; overflow: hidden;"><div style="border:1px solid #cccccc; padding: 3px; background-color:#f9f9f9; width: 355px;">http://xs68.xs.to/pics/06075/worf-shadow-subs-manysubsinone.png</div></div>
<div style="width: auto; overflow: hidden;"><div style="border:1px solid #cccccc; padding: 3px; background-color:#f9f9f9; width: 355px;">http://xs68.xs.to/pics/06075/worf-shadow-subs-manysubsinone.png</div></div>
As you can see here, the actual raws do not need shadow-files to encompass them, since in this case it's one raw for each episode.
As you can see here, the actual RAWs do not need shadow-files to encompass them, since in this case it's one raw for each episode.


Still, when it comes to split-episodes the "original" verision of the shadow-files is necessary - you need the abstraction layer there.
Still, when it comes to split-episodes the "original" verision of the shadow-files is necessary - you need the abstraction layer there.
Line 48: Line 50:
:<pelican> A further consideration: subtitles must be available even if no other file for the episode or episodes they represent is
:<pelican> A further consideration: subtitles must be available even if no other file for the episode or episodes they represent is
</tt></div>
</tt></div>
This would take care of this concern that pelican had. Atleast in my opinion. He did have a rather different approach to the problem than what is described here, and as I don't understand it fully, I can only hope that he will describe it properly himself.
This would take care of this concern that pelican had. At least in my opinion. He did have a rather different approach to the problem than what is described here, and as I don't understand it fully, I can only hope that he will describe it properly himself.


----
----


Oh wait, I just realized that I didn't explain this part properly: "<tt>and be assigned to however many other files is necessary.</tt>"<br>
Oh wait, I just realized that I didn't explain this part properly: "<tt>and be assigned to however many other files is necessary.</tt>"<br>
What I mean is that ''other RAWs for the same episode can usually use the same subs'', so the files would haveto be able to have many "parent" files, and the same subs should also be listed properly under the actual episode for those who just want the subs and don't need/already have/only want to steal/whatever can see that the episode actually has separate subs.
What I mean is that ''other RAWs for the same episode can usually use the same subs'', so the files would have to be able to have many "parent" files, and the same subs should also be listed properly under the actual episode for those who just want the subs and don't need/already have/only want to steal/whatever can see that the episode actually has separate subs.
</div>
</div>
:<small>hope this is actually helpful to some ppl, so I'm not just blowing off steam on a non-issue >_<</small> ... I do feel that this needs a more proper going through, if exp agrees or not, I have no clue. It also seems unclear (atleast to me) whether exp has decided on an actual design when it comes to shadow-files or if that is still up for discussion; there seems to be a few people who still has something to say about that, but atleast PetriW did seem to [http://www.anidb.net/forum/viewtopic.php?p=7294#7294 describe] the first of the shadow-files versions described here.
:{{small90|Hope this is actually helpful to some ppl, so I'm not just blowing off steam on a non-issue >_<}} ... I do feel that this needs a more proper going through, if exp agrees or not, I have no clue. It also seems unclear (at least to me) whether exp has decided on an actual design when it comes to shadow-files or if that is still up for discussion; there seems to be a few people who still has something to say about that, but at least PetriW did seem to [http://www.anidb.net/forum/viewtopic.php?p=7294#7294 describe] the first of the shadow-files versions described here.
<div style="text-align: right;">--[[User:Suppy|Suppy]] 02:20, 18 February 2006 (CET)</div>
<div style="text-align: right;">--[[User:Suppy|Suppy]] 02:20, 18 February 2006 (CET)</div>
<div style="border: 1px solid; background-color: #f2f2f2; padding: 5px;">
this is my first wiki entry. I hope i stay within all rules. And i hope i'm not talking about something already discussed and shredded...<br>
Ok, i thought about this too. As i see it, there'll be a major change in db structure, so i just started without thinking of the current structure. The main idea is to just insert another layer between current episodes and files.<br>
<br>
I'll explain it on a few examples:<br>
1) normal case:<br>
a series has several episodes, each of these episodes has several releases, which consist each of one file.<br>
<br>
<pre>
[series]--->[episode1]--->[release]--->[file]
        |->[epsiode2]--->[release]--->[file]
        |->[epsiode3]--->[release]--->[file]
        \->[epsiode4]--->[release]--->[file]
</pre>
2) multiple files for one movie/episode:<br>
<pre>
                                  /->[file1]
                                  /
[series]--->[episode]--->[release]--->[file2]
                                  \
                                  \->[file3]
</pre>
3) one file containing several episodes:<br>
<pre>
            /->[episode1]-\
          /              \
[series]------>[episode2]----->[release]--->[file]
          \              /
            \->[episode3]-/
</pre>
4) all in the mix
<pre>
[series]--->[episode1]--->[release]--->[file1]
        |            |            |->[file2]
        |            |            \->[file3]
        |            |
        |            |->[release]--->[file]
        |            \->[release]--->[generic]
        |
        |->[episode2]--->[release]--->[generic]
        |            \
        |            |->[release]--->[file]
        |            /
        |->[episode3]--->[release]--->[generic]
        |            \
        |            |            /->[file2]
        |            |->[release]--->[file1]
        |            |            \->[file3]
        |            /
        |->[episode3]--->[release]--->[generic]
</pre>
i don't think it'd be a good idea if too many exceptions or restrictions are made. I'd also not try to calculate any percentages of how much a file covers an episode. this all leads to more complexity and thus to more "features". if all files from a release are watched, the linked episodes are watched too ... <br>
<br>
following information should be saved here:<br>
series: same as now<br>
episodes: same as now<br>
releases: everything thats now stored in the file entry but that's the same for all files of a release (e.g. subbing group, quality, bitrate, resolution, etc.)<br>
files: same as now without what's stored in the releases layer plus what additional files exist for this file (e.g. external subtitles, nfo, etc.)<br>
<br>
about subtitle files:<br>
i don't think it's a good idea to allow them to be connected to more than one file. for the cases, where several raw eps exist where the timing and evering fits perfectly i sugest to add it twice. if this is a bad idea, i'd say additional files like subtitle files, nfo files etc. should be stored in a separate table. these files can then be added to each file where they fit. i wouldn't connect them with each raw of an episode by default. I'd only allow adding them manually for each file where they match
</div><div style="text-align: right;">--[[User:Tomynocker|Tomynocker]] 17:38, 2 March 2006 (CET)</div>


==Grouped shadow-files==
==Grouped shadow-files==
<div style="border: 1px solid; background-color: #f2f2ff; padding: 5px;">
<div style="border: 1px solid; background-color: #f2f2ff; padding: 5px;">
pelican later came with a rather different suggestion...<br>
pelican later came with a rather different suggestion...<br>
<small>paraphrased...</small>
{{small90|paraphrased...}}
<div style="border: 1px solid gray; background-color: #f9f9f9;"><tt>
<div style="border: 1px solid gray; background-color: #f9f9f9;"><tt>
:<pelican> The problem with subtitle association isn't the db structure (I could draw it in a minute if, um... if I could be bothered), it's the interface
:<pelican> The problem with subtitle association isn't the db structure (I could draw it in a minute if, um... if I could be bothered), it's the interface
Line 138: Line 71:
:<pelican> Have fun.
:<pelican> Have fun.
:<pelican> Oh, and even if exp implements shadow files (unlikely), subtitle association on top of that is probably more work than he's up for
:<pelican> Oh, and even if exp implements shadow files (unlikely), subtitle association on top of that is probably more work than he's up for
:<pelican> In case you're interested, the best structure is probably to have episode collections, which uniquely represent sets of episodes that shadow files use, put another layer on top of that representing same timing of an episode set and have subtitles act the same as normal files; you can group shadow files under one timing group with a mod change or (since this is an imaginary anidb without the restrictions of the current) creq
:<pelican> In case you're interested, the best structure is probably to have episode collections, which uniquely represent sets of episodes that shadow files use, put another layer on top of that representing same timing of an episode set and have subtitles act the same as normal files; you can group shadow files under one timing group with a mod change or (since this is an imaginary AniDB without the restrictions of the current) creq
:<pelican> ...hope that line wasn't too long for the server
:<pelican> ...hope that line wasn't too long for the server
:<pelican> All shadow files are then associated with a timing group
:<pelican> All shadow files are then associated with a timing group
Line 187: Line 120:
We'll begin with the basic ''Separate subtitles for one episode'' example:
We'll begin with the basic ''Separate subtitles for one episode'' example:
<div style="width: auto; overflow: hidden;"><div style="border:1px solid #cccccc; padding: 3px; background-color:#f9f9f9; width: 654px;">http://xs69.xs.to/pics/06082/groups-subs-normal.png</div></div>
<div style="width: auto; overflow: hidden;"><div style="border:1px solid #cccccc; padding: 3px; background-color:#f9f9f9; width: 654px;">http://xs69.xs.to/pics/06082/groups-subs-normal.png</div></div>
Here you can have any number of raw files and subtitle files, as long as they're timed "for eachother".
Here you can have any number of RAW files and subtitle files, as long as they're timed "for each other".


There is one problem that I haven't been able to figure out how to solve with this method, and that is: '''How to fix the "many subs in one file" problem.''' I asked pelican about it:<br>
There is one problem that I haven't been able to figure out how to solve with this method, and that is: '''How to fix the "many subs in one file" problem.''' I asked pelican about it:<br>
<small>paraphrased (boldness added for focusing your eyes)</small>
{{small90|paraphrased (boldness added for focusing your eyes)}}
<div style="border: 1px solid gray; background-color: #f9f9f9;"><tt>
<div style="border: 1px solid gray; background-color: #f9f9f9;"><tt>
:<hrm> hmm, with your episode sets and timed episode sets - how do you solve the "many subs in one file" problem illustrated here: http://wiki.anidb.info/w/Image:Current-subs-manysubsinone.png  
:<hrm> hmm, with your episode sets and timed episode sets - how do you solve the "many subs in one file" problem illustrated here: [[Image:Current-subs-manysubsinone.png]]
:<hrm> I'm trying to illustrate the example of having a rar or zip file that contains many subs for more than one episode
:<hrm> I'm trying to illustrate the example of having a rar or zip file that contains many subs for more than one episode
[...]
[...]
Line 219: Line 152:


<div style="text-align: right;">--[[User:Worf|Worf]] 00:56, 22 February 2006 (CET)</div>
<div style="text-align: right;">--[[User:Worf|Worf]] 00:56, 22 February 2006 (CET)</div>
==Current DB Structure==
Well, I thought it might benefit this topic to have a short description of the current DB structure. The relations look like this: (>-< N:M, -< 1:N, >- N:1)
Anime -< Episode -< File -< Stream
However, there are two additional relations as workarounds:
File >-< File
To allow things like File-to-Subtitle relations, Files which form a unit or files which represent a newer version of another file.
Episode >-< File
To allow files which span multiple episodes or files which span only part of an episode. The main relation remains Episode -< File mainly for performance reasons. Only a small subset of the files in the DB belong to more than one EP.
--[[User:Exp|Exp]] 03:56, 3 March 2006 (CET)

Latest revision as of 09:39, 14 May 2009

Multi-level shadow-files

<worf> aren't subtitle files supposed to be a sub-file of the actual video files?
<worf> rather than being on the same level as them
<hrm> you might have a point there worf
<pelican> Linking subtitles to just one video file is stupid; any number makes more sense, but I'm not terribly keen on it anyway
<hrm> worf, but that would make the regular files a kind of shadow-files as well... I assumed we would keep with the current structure of the regular files, and just add the abstraction-layer of the shadow-files
<hrm> ah yeah, what pelican said is also a point
<worf> thats something i cant comment on
<pelican> Normally if you implement this kind of abstraction layer, you make it universal
<pelican> It's how relational databases work
<worf> yeah
[...]
<pelican> So basically... yes, you can restrict shadow files to files which need them, but it's probably going to be less efficient and it's certainly more code

I did a little thinking on this, and I think worf's suggestion holds merit. Consider the following scenario: Two split-episode files are raw, and they have subs. Currently this would look like this:

With my current suggestion for shadow-files, it would be like the following picture:

With worf's suggestion it would be like this:

And I have to say that worf's is the most intuitive of them. Basically every single file in the DB, would in and of themselves actually be shadow-files, having the ability to have sub-files and/or parent-files. Note that I use plural here - restricting parent- and sub-files to only one per file would be bad. RAWs would only be able to have one single subtitle file, while in actuality, there are lots of RAWs that have more than one version of a subtitle file, and also, subtitle files would only be able to be associated with one single RAW, while they usually work with more than one (not to mention the "separate subtitles for many episodes" case (see below)). Note also that if ordinary files were in fact shadow-files, this would make the extra abstraction layer found in this:

be unnecessary.

Worf's version does however raise the question about what to do with separate subtitles for many episodes, and I assume the result would be something like this:

As you can see here, the actual RAWs do not need shadow-files to encompass them, since in this case it's one raw for each episode.

Still, when it comes to split-episodes the "original" verision of the shadow-files is necessary - you need the abstraction layer there.

I wasn't exactly sure where to put this discussion, but I figured this would be the best place for it. Maybe I should have posted to forum or something instead, but imo, the wiki works better for this.
--Suppy 03:17, 17 February 2006 (CET)

Also another case which would be relatively weird if you didn't have multi-layered support: .sub/.idx combinations for split episode files. Let me illustrate: Currently:

With original shadow-files:

With worf's shadow-files:

As you can see, this would require that every file be able to have files under it and be assigned to however many other files is necessary.

<pelican> A further consideration: subtitles must be available even if no other file for the episode or episodes they represent is

This would take care of this concern that pelican had. At least in my opinion. He did have a rather different approach to the problem than what is described here, and as I don't understand it fully, I can only hope that he will describe it properly himself.


Oh wait, I just realized that I didn't explain this part properly: "and be assigned to however many other files is necessary."
What I mean is that other RAWs for the same episode can usually use the same subs, so the files would have to be able to have many "parent" files, and the same subs should also be listed properly under the actual episode for those who just want the subs and don't need/already have/only want to steal/whatever can see that the episode actually has separate subs.

Hope this is actually helpful to some ppl, so I'm not just blowing off steam on a non-issue >_< ... I do feel that this needs a more proper going through, if exp agrees or not, I have no clue. It also seems unclear (at least to me) whether exp has decided on an actual design when it comes to shadow-files or if that is still up for discussion; there seems to be a few people who still has something to say about that, but at least PetriW did seem to describe the first of the shadow-files versions described here.
--Suppy 02:20, 18 February 2006 (CET)

Grouped shadow-files

pelican later came with a rather different suggestion...
paraphrased...

<pelican> The problem with subtitle association isn't the db structure (I could draw it in a minute if, um... if I could be bothered), it's the interface
<pelican> Things are already complicated by having shadow files which may represent multiple episodes
<pelican> A further consideration: subtitles must be available even if no other file for the episode or episodes they represent is
<pelican> They cannot be dependent
<pelican> Have fun.
<pelican> Oh, and even if exp implements shadow files (unlikely), subtitle association on top of that is probably more work than he's up for
<pelican> In case you're interested, the best structure is probably to have episode collections, which uniquely represent sets of episodes that shadow files use, put another layer on top of that representing same timing of an episode set and have subtitles act the same as normal files; you can group shadow files under one timing group with a mod change or (since this is an imaginary AniDB without the restrictions of the current) creq
<pelican> ...hope that line wasn't too long for the server
<pelican> All shadow files are then associated with a timing group
<pelican> The structure is (episode)>-<(episode set)-<(timed episode set)-<(shadow file)-<(file)
<pelican> Where >-< is many-many, -< one-many, and - would be one-one if I used it
<pelican> Current structure is (episode)-<(file)
<pelican> The episode sets are really just to allow us to quickly find timed episode sets of the same episodes
<pelican> This structure would require 6 tables, since many to many relationships need a relation table

[...]

<pelican> Ah, but shadow files group files together which are components of one shadow file; subtitles aren't components of another file, they're peripheral
<pelican> Having sub files under a/v is just wrong
<hrm> they're peripheral yes. but they're peripherals _to_ something. just like your mouse is a peripheral to your computer... you wouldn't go out and buy a mouse without either already having a computer or buying a computer at the same time.
<pelican> They're peripheral to the episodes
<pelican> Not to the files
<hrm> they are peripheral to the episodes, yes. but they are _also_ peripheral to the files.
<pelican> Read my suggested relations and you should see
<pelican> No, they are parallel to the files
<pelican> Files which represent the same group of episodes timed the same are interchangeable and can be used together where appropriate
<pelican> As with subtitles and a/v
<hrm> well, I didn't really get the whole thing with timed groups exactly. I would ask for more elaboration but I'm in the middle of watching a movie.
<pelican> It's just a set for which all (shadow) files would run in sync
<pelican> And thus in which you can apply sub files to a/v
<hrm> if you can explain to me how that would be done for something like this: http://xs68.xs.to/pics/06076/worf-shadow-subs-split-subandidx.png I'd appreciate it. that is supposed to be a 2cd release of a movie with accompanying .sub & .idx subtitle files.
<pelican> The .sub and .idx form a shadow file
<pelican> That shadow file is in the same timing group as raws whose timing it matches
<hrm> ah, I see. the way that would be handled in my system is that the .sub and .idx shadowfile would also be added to every other raw whose timing they match, as well as directly to the episode. .... it's just that for simplicity's sake, I only illustrated a single chain
<pelican> Best to only have one thing in one place
<pelican> And this automates association of them with other raws with the same timing
<hrm> yeah, that is a good point

In case you didn't exactly get all of that (it took a while before I did too ^^ (I'm still not sure I do ;D)), I'll do my best to illustrate it in a good way here ... Let's begin with the standard One file for One episode example, to get us started:

Now... in case you don't understand why all the fuss is necessary here (I mean: what group - it's just one single file! just one single episode!) .. it's because of uniformity, so different code isn't needed for different cases ... besides, those layers would be all but invisible to the actual user. You'll see why they're actually needed in the next couple of examples.

Let's begin with the multi-episode file example:

This should show you why the episode group is needed. Now let's move on to the timed group, which has more examples as to why it is needed.

Split-episode files:

I am not certain here whether the division should take place between (as I've illustrated) shadow-file and file, or if it should be between timed group and shadow-file. I'll haveto ask pelican to explain that more thoroughly. I believe I have guessed wrong here, but it was the way I did it, and it's too much of a bother to fix it now ;). If pelican says I'm wrong I'll do it however.

Moving on to the separate subs ...

We'll begin with the basic Separate subtitles for one episode example:

Here you can have any number of RAW files and subtitle files, as long as they're timed "for each other".

There is one problem that I haven't been able to figure out how to solve with this method, and that is: How to fix the "many subs in one file" problem. I asked pelican about it:
paraphrased (boldness added for focusing your eyes)

<hrm> hmm, with your episode sets and timed episode sets - how do you solve the "many subs in one file" problem illustrated here:
<hrm> I'm trying to illustrate the example of having a rar or zip file that contains many subs for more than one episode

[...]

<pelican> It doesn't deal with it; you can't expect shadow files to fix everything
<pelican> There is another structure that could work
<pelican> Move the timed structure to individual episodes
<pelican> However, this means that a single subtitles file which covers multiple episodes won't necessarily be timed correctly and it introduces further complications
<hrm> darn, that's what I feared you would say =/
<pelican> Your original idea of associating files and subtitles doesn't have this problem of course, but it's a fucking mess
<pelican> I wouldn't let it into any db I'd written
<pelican> (exp might, but he's never going to write any of this so that's not an issue)

"you can't expect shadow files to fix everything" - darn, I was hoping it would =/

This way of doing things doesn't solve everything, however it is a much cleaner way of fixing the shadow-files, and it solves most of the problems we currently have. A very viable option.
--Suppy 10:46, 21 February 2006 (CET)


This draft should contain all possible cases but one. Split files (Episode 0x Part 1 & Episode 0x Part 2) with both .sub and .idx subtitle files. We'd have to group the two split files under one shadow file but what to do with the four subtitle files? Grouping all four of them under one shadow file under the same timed group wouldn't be practical as we couldn't be sure which pair belongs to which split file.

--Worf 00:56, 22 February 2006 (CET)

Current DB Structure

Well, I thought it might benefit this topic to have a short description of the current DB structure. The relations look like this: (>-< N:M, -< 1:N, >- N:1)

Anime -< Episode -< File -< Stream

However, there are two additional relations as workarounds:

File >-< File To allow things like File-to-Subtitle relations, Files which form a unit or files which represent a newer version of another file.

Episode >-< File To allow files which span multiple episodes or files which span only part of an episode. The main relation remains Episode -< File mainly for performance reasons. Only a small subset of the files in the DB belong to more than one EP.

--Exp 03:56, 3 March 2006 (CET)