1,633
edits
m (client commands in source boxes) |
mNo edit summary |
||
(One intermediate revision by one other user not shown) | |||
Line 4: | Line 4: | ||
VERSION 2 | VERSION 2 | ||
Servers: | Servers: | ||
* dedicated Java matching server daemon TCP on anidb3 (sig) (does not accept client requests directly) | * dedicated Java matching server daemon TCP on anidb3 (sig) (does not accept client requests directly) | ||
* dedicated Java redirector/load balancer daemon TCP and UDP on anidb2 (main/api) (UDP for single queries, TCP for batch runs) | * dedicated Java redirector/load balancer daemon TCP and UDP on anidb2 (main/api) (UDP for single queries, TCP for batch runs) | ||
Clients: | Clients: | ||
* cronjob on anidb2 | * cronjob on anidb2 | ||
* clients on user systems | * clients on user systems | ||
== General Workflow == | == General Workflow == | ||
=== Workflow Draft === | === Workflow Draft === | ||
==== Involved Parties ==== | ==== Involved Parties ==== | ||
* client ( | * client (Avdump), locally on user machine | ||
* main server (anidb2), keeps file meta data includes fingerprints | * main server (anidb2), keeps file meta data includes fingerprints | ||
* matching redirector (anidb2), a simple load balancer which redirects all requests to the corresponding matching server(s) (MATCH goes to all, STORE to one), TCP and UDP interface (UDP for external queries, TCP for cron job/batch queries) | * matching redirector (anidb2), a simple load balancer which redirects all requests to the corresponding matching server(s) (MATCH goes to all, STORE to one), TCP and UDP interface (UDP for external queries, TCP for cron job/batch queries) | ||
* matching server(s) (anidb3/sig server), Java standalone app does all the fingerprint<->fingerprint matching, TCP interface | * matching server(s) (anidb3/sig server), Java standalone app does all the fingerprint<->fingerprint matching, TCP interface | ||
==== Workflow - Matching ==== | ==== Workflow - Matching ==== | ||
===== Synchronous Part ===== | ===== Synchronous Part ===== | ||
* user runs client on his local audio files | * user runs client on his local audio files | ||
Line 34: | Line 27: | ||
** optional intermediate step to speedup client processing: calculate content hash first and generate and submit the fingerprint only if the content hash is unknown to AniDB or no fingerprint is listed on AniDB for that file | ** optional intermediate step to speedup client processing: calculate content hash first and generate and submit the fingerprint only if the content hash is unknown to AniDB or no fingerprint is listed on AniDB for that file | ||
* main server does not do any processing on the fingerprint and simply stores it in ostfiletb | * main server does not do any processing on the fingerprint and simply stores it in ostfiletb | ||
===== Asynchronous Part ===== | ===== Asynchronous Part ===== | ||
Line 61: | Line 53: | ||
* main server uses matching data to support manual file<->song matching via the web interface | * main server uses matching data to support manual file<->song matching via the web interface | ||
** the user will be able select the cut-off-point for the matching value on-the-fly in order to reduce false-positives or increase recall | ** the user will be able select the cut-off-point for the matching value on-the-fly in order to reduce false-positives or increase recall | ||
==== Workflow - Deletion ==== | ==== Workflow - Deletion ==== | ||
Line 72: | Line 63: | ||
* the matching servers will delete all local data for the given ofid | * the matching servers will delete all local data for the given ofid | ||
* the results are collated by the matching redirector and returned to the main server's cron job | * the results are collated by the matching redirector and returned to the main server's cron job | ||
== Possible Extension == | == Possible Extension == | ||
Line 80: | Line 70: | ||
* same commands as available via TCP (used by the anidb2 cronjob) also available via UDP for use by other clients. | * same commands as available via TCP (used by the anidb2 cronjob) also available via UDP for use by other clients. | ||
** i.e. to allow lookups by foosic fingerprint. For that a client would first contact the UDP API (anidb2) with the content hash and if the content hash is unknown to AniDB it would send the fingerprint to the matching redirector (anidb2), which would delegate to the matching servers (anidb3), to get one or more ostfile id(s) and then use those to query song data from the UDP API. | ** i.e. to allow lookups by foosic fingerprint. For that a client would first contact the UDP API (anidb2) with the content hash and if the content hash is unknown to AniDB it would send the fingerprint to the matching redirector (anidb2), which would delegate to the matching servers (anidb3), to get one or more ostfile id(s) and then use those to query song data from the UDP API. | ||
*** this is not meant for | *** this is not meant for Avdump, but it might be interesting for direct integration into player software, i.e. a winamp/amarok plugin, would work somewhat like the already available musicbrainz plugins. | ||
== General Considerations == | == General Considerations == | ||
Line 103: | Line 93: | ||
== Protocol == | == Protocol == | ||
=== Broken Clients === | === Broken Clients === | ||
* we probably want to require a client string and client version in every query (similar to UDP API) to be able to ban badly broken clients, should the need arise someday. | * we probably want to require a client string and client version in every query (similar to UDP API) to be able to ban badly broken clients, should the need arise someday. | ||
=== Protocol Draft === | === Protocol Draft === | ||
Every query should contain as additional parameters: | Every query should contain as additional parameters: | ||
client={str client name}&clientver={int client version} | client={str client name}&clientver={int client version} | ||
Access to the STORE command will be limited to the main server's cron job by this method. | Access to the STORE command will be limited to the main server's cron job by this method. | ||
==== Querying a foosic fingerprint (don't add if unknown) ==== | ==== Querying a foosic fingerprint (don't add if unknown) ==== | ||
Used by: | Used by: | ||
* external clients | * external clients | ||
* main server's cron job | * main server's cron job | ||
* matching redirector (forwarded) | * matching redirector (forwarded) | ||
Client: | Client: | ||
MATCH ofid={int4 ostfile id}&foosic={str ascii hex representation of fingerprint}[&store=1] | MATCH ofid={int4 ostfile id}&foosic={str ascii hex representation of fingerprint}[&store=1] | ||
: The <tt>store=1</tt> parameter is filtered out and interpreted by the matching redirector, only the main server's cron job is allowed to set <tt>store=1</tt>. | : The <tt>store=1</tt> parameter is filtered out and interpreted by the matching redirector, only the main server's cron job is allowed to set <tt>store=1</tt>. | ||
Server Reply: | Server Reply: | ||
Line 141: | Line 125: | ||
if the ofid being queried is stored on that specific matching server | if the ofid being queried is stored on that specific matching server | ||
:* the matching redirector will filter 201 replies, a normal client will never see them. | :* the matching redirector will filter 201 replies, a normal client will never see them. | ||
* No matchings found | * No matchings found | ||
Line 154: | Line 137: | ||
if the ofid being queried is stored on that specific matching server | if the ofid being queried is stored on that specific matching server | ||
:* the matching redirector will filter 201 replies, a normal client will never see them. | :* the matching redirector will filter 201 replies, a normal client will never see them. | ||
==== Submitting a new foosic fingerprint ==== | ==== Submitting a new foosic fingerprint ==== | ||
Used by: | Used by: | ||
* matching redirector, access is restricted | * matching redirector, access is restricted | ||
Client: | Client: | ||
STORE ofid={int4 ostfile id}&foosic={str ascii hex representation of fingerprint} | STORE ofid={int4 ostfile id}&foosic={str ascii hex representation of fingerprint} | ||
Server Reply: | Server Reply: | ||
Line 172: | Line 151: | ||
* Fingerprint was already in DB | * Fingerprint was already in DB | ||
: 310 ALREADY STORED | : 310 ALREADY STORED | ||
==== Submitting a new foosic fingerprint ==== | ==== Submitting a new foosic fingerprint ==== | ||
Used by: | Used by: | ||
* main server's cron job, access is restricted | * main server's cron job, access is restricted | ||
* matching redirector (forwarded), access is restricted | * matching redirector (forwarded), access is restricted | ||
Client: | Client: | ||
DELETE ofid={int4 ostfile id}[&{int2 ident of matching server this fingerprint is stored on}] | DELETE ofid={int4 ostfile id}[&{int2 ident of matching server this fingerprint is stored on}] | ||
Server Reply: | Server Reply: | ||
Line 191: | Line 166: | ||
* Fingerprint was not in DB | * Fingerprint was not in DB | ||
: 320 NOT FOUND | : 320 NOT FOUND | ||
==== Query the current server load/utilization ==== | ==== Query the current server load/utilization ==== | ||
Used by: | Used by: | ||
* matching redirector, access is restricted | * matching redirector, access is restricted | ||
Client: | Client: | ||
LOADSTAT | LOADSTAT | ||
Server Reply: | Server Reply: |
edits