OstDB DEV Foosic: Difference between revisions

VERSION 2

Servers:

* dedicated Java matching server daemon TCP on anidb3 (sig) (does not accept client requests directly)

* dedicated Java redirector/load balancer daemon TCP and UDP on anidb2 (main/api) (UDP for single queries, TCP for batch runs)

Clients:

* cronjob on anidb2

* clients on user systems

== General Workflow ==

=== Workflow Draft ===

==== Involved Parties ====

* client (avdump), locally on user machine

* matching redirector (anidb2), a simple load balancer which redirects all requests to the corresponding matching server(s) (MATCH goes to all, STORE to one), TCP and UDP interface (UDP for external queries, TCP for cron job/batch queries)

* matching server(s) (anidb3/sig server), Java standalone app does all the fingerprint<->fingerprint matching, TCP interface

==== Workflow - Matching ====

===== Synchronous Part =====

* user runs client on his local audio files

** optional intermediate step to speedup client processing: calculate content hash first and generate and submit the fingerprint only if the content hash is unknown to AniDB or no fingerprint is listed on AniDB for that file

* main server does not do any processing on the fingerprint and simply stores it in ostfiletb

===== Asynchronous Part =====

* main server uses matching data to support manual file<->song matching via the web interface

** the user will be able select the cut-off-point for the matching value on-the-fly in order to reduce false-positives or increase recall

==== Workflow - Deletion ====

* the matching servers will delete all local data for the given ofid

* the results are collated by the matching redirector and returned to the main server's cron job

== Possible Extension ==

== Protocol ==

=== Broken Clients ===

* we probably want to require a client string and client version in every query (similar to UDP API) to be able to ban badly broken clients, should the need arise someday.

=== Protocol Draft ===

Every query should contain as additional parameters:

  client={str client name}&clientver={int client version}

Access to the STORE command will be limited to the main server's cron job by this method.

==== Querying a foosic fingerprint (don't add if unknown) ====

Used by:

* external clients  

* main server's cron job

* matching redirector (forwarded)

Client:

  MATCH ofid={int4 ostfile id}&foosic={str ascii hex representation of fingerprint}[&store=1]

: The <tt>store=1</tt> parameter is filtered out and interpreted by the matching redirector, only the main server's cron job is allowed to set <tt>store=1</tt>.

Server Reply:

if the ofid being queried is stored on that specific matching server

:* the matching redirector will filter 201 replies, a normal client will never see them.

* No matchings found

if the ofid being queried is stored on that specific matching server

:* the matching redirector will filter 201 replies, a normal client will never see them.

==== Submitting a new foosic fingerprint ====

Used by:

* matching redirector, access is restricted

Client:

  STORE ofid={int4 ostfile id}&foosic={str ascii hex representation of fingerprint}

Server Reply:

* Fingerprint was already in DB

: 310 ALREADY STORED

==== Submitting a new foosic fingerprint ====

Used by:

* main server's cron job, access is restricted

* matching redirector (forwarded), access is restricted

Client:

  DELETE ofid={int4 ostfile id}[&{int2 ident of matching server this fingerprint is stored on}]

Server Reply:

* Fingerprint was not in DB

: 320 NOT FOUND

==== Query the current server load/utilization ====

Used by:

* matching redirector, access is restricted

Client:

  LOADSTAT

Server Reply: