1,633
edits
Auto-creqing: Difference between revisions
m
no edit summary
Epoximator (talk | contribs) No edit summary |
mNo edit summary |
||
| Line 12: | Line 12: | ||
The "new" system, which is used by avdump, is not part of any API and not documented anywhere. (The server side part is however related the UDP API server). Instead of checking against AniDB data and issue creqs directly when necessary, avdump just sends the information (per file) to the server without any regard. It is entirely up to the server to decide what to do with the raw data that is received. This new approach was implemented to shift the control from the clients to the server. | The "new" system, which is used by avdump, is not part of any API and not documented anywhere. (The server side part is however related the UDP API server). Instead of checking against AniDB data and issue creqs directly when necessary, avdump just sends the information (per file) to the server without any regard. It is entirely up to the server to decide what to do with the raw data that is received. This new approach was implemented to shift the control from the clients to the server. | ||
The different parts of the system is; avdump, AVMF server, AVMF service, | The different parts of the system is; avdump, AVMF server, AVMF service, auto-granter and the [http://anidb.net/perl-bin/animedb.pl?show=avmf avmf page]. | ||
====Avdump==== | ====Avdump==== | ||
| Line 22: | Line 22: | ||
====AVMF server==== | ====AVMF server==== | ||
Runs in the same process as the UDP API Server, written in | Runs in the same process as the UDP API Server, written in Java. Is supposed to be very light and simple to minimize possible issues (downtime). | ||
# Accept/read new AVMF package. | # Accept/read new AVMF package. | ||
# Decrypt, decompress the package. If it should fail the package will just be disregarded. | # Decrypt, decompress the package. If it should fail the package will just be disregarded. | ||
| Line 31: | Line 31: | ||
====AVMF service==== | ====AVMF service==== | ||
Runs every 5 min in the same process as the UDP API Server, written in | Runs every 5 min in the same process as the UDP API Server, written in Java. | ||
# Query new AVMF dumps which has been mapped to at least 10 minutes old files (grace period). | # Query new AVMF dumps which has been mapped to at least 10 minutes old files (grace period). | ||
## Check the metadata against the mapped AniDB file. | ## Check the metadata against the mapped AniDB file. | ||
| Line 38: | Line 38: | ||
====Autogranter==== | ====Autogranter==== | ||
Cron job written in | Cron job written in Perl, actually a part of the ''old system''. The main reasons for a separate granter is to have a double sanity check and enough delay to notice (and halt) possible errors. | ||
# Query automatic change requests older than 24h. | # Query automatic change requests older than 24h. | ||
## Check that the request is sane and untouched (by owner/mod). | ## Check that the request is sane and untouched (by owner/mod). | ||
| Line 53: | Line 53: | ||
===Conflicting data=== | ===Conflicting data=== | ||
The main challenge, for both systems, lies in the fact that extracting metadata is not an exact | The main challenge, for both systems, lies in the fact that extracting metadata is not an exact science, or at least not trivial: Different clients are bound to come up with different data which means that the system has to handle conflicting data one way or the other. This issue has basically been evaded by only having one trusted client (and version) which is Avdump (.31) ATM. AOM does still issue autocreqs but only hash sums and duration are included and only for files that are not already verified by Avdump. Data from Avdump will override data from AOM if they should differ (never happened for hash sums). | ||
Avdump is supposed to be a shared library that all clients can use for | Avdump is supposed to be a shared library that all clients can use for auto-creqing, though. It is however not clear how it should be done yet. | ||
==AVMF states== | ==AVMF states== | ||
| Line 70: | Line 70: | ||
| used || A file is considered ''verified'' if it has one ''used'' dump. There can only be one dump with this state for each file at once. In the process of changing the state of a dump to ''used'', automatic change requests will be generated if needed. | | used || A file is considered ''verified'' if it has one ''used'' dump. There can only be one dump with this state for each file at once. In the process of changing the state of a dump to ''used'', automatic change requests will be generated if needed. | ||
|- | |- | ||
| deprecated || The dump was in use at some point, but has been automatically replaced by an other. Dumps of newer versions (produced by newer versions of avdump) will automatically replace dumps of older versions ''unless it requires new creqs''. Dumps with this state might be purged | | deprecated || The dump was in use at some point, but has been automatically replaced by an other. Dumps of newer versions (produced by newer versions of avdump) will automatically replace dumps of older versions ''unless it requires new creqs''. Dumps with this state might be purged regularly at some point, but that is not enabled ATM. | ||
|- | |- | ||
| candidate || The dump is not used because the related file is already verified by another dump of an older version of avdump. The service will never issue creqs for files that has already been verified. A moderator can however delete the ''used'' dump (or smite it) and then ''reset'' the candidate if needed. | | candidate || The dump is not used because the related file is already verified by another dump of an older version of avdump. The service will never issue creqs for files that has already been verified. A moderator can however delete the ''used'' dump (or smite it) and then ''reset'' the candidate if needed. | ||
| Line 78: | Line 78: | ||
| stalled || The dump is stalled due a pending change request of the related file. Stalled dumps will be reset to ''new'' automatically after some time. | | stalled || The dump is stalled due a pending change request of the related file. Stalled dumps will be reset to ''new'' automatically after some time. | ||
|- | |- | ||
| incoherent || The dump is not trusted, and thus not used, because the track bitrates does not match up to the file size. Too much overhead in other words. The threshold is 6% | | incoherent || The dump is not trusted, and thus not used, because the track bitrates does not match up to the file size. Too much overhead in other words. The threshold is 6% ATM. | ||
|- | |- | ||
| exception || The AVMF server threw an exception when handling an incoming dump. | | exception || The AVMF server threw an exception when handling an incoming dump. | ||
| Line 88: | Line 88: | ||
===Manual states=== | ===Manual states=== | ||
These states can only be set by moderators. They should be avoided as far as | These states can only be set by moderators. They should be avoided as far as possible; keeping the dump ''used'' is preferred. | ||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
! State !! Description | ! State !! Description | ||
|- | |- | ||
| reset || A moderator wants the dump to be | | reset || A moderator wants the dump to be re-handled by the AVMF service, usually because he/she wants to unlock the related file temporarily. Lifecycle: new -> used -> reset -> new -> used. It would be better to improve/change the interface (regarding locks) instead of using this state. | ||
|- | |- | ||
| smitten || The dump is considered invalid by a moderator. This state can be used by developers to pick up and handle issues with avdump. It is better to use [[avdump issues]] to keep track of these dumps, though. | | smitten || The dump is considered invalid by a moderator. This state can be used by developers to pick up and handle issues with avdump. It is better to use [[avdump issues]] to keep track of these dumps, though. | ||