OstDB DEV Foosic: Difference between revisions

Protocol for foosic client<->server communication via UDP and TCP

VERSION 1

Client: cronjob/daemon on anidb2

* matching redirector (anidb2), a simple load balancer which redirects all requests to the corresponding matching server(s) (IDENT goes to all, STORE to one), TCP and UDP interface (UDP for external queries, TCP for cron job/batch queries)

* user runs client on local audio files

* cronjob/daemon on main server regularly checks for newly added foosic fingerprints and sends them, together with the ostfile id to the matching redirector via TCP with store set to 1

* matching redirector forwards the IDENT query to all matching servers via TCP

** TCP connections to all matching servers should be kept alive inbetween queries (prevent TCP connection handshake overhead)

* matching servers try to match the fingerprint to all potentially interesting fingerprints in their database

** post-match filtering: via hard cut-off-point for matching, this will definitely be >=0,50 anything lower would be useless. It will be possible to increase this value without problems at any point in time. However, reducing it would be a problem. We might therefore want to just start with 0,50 pr 0,60 and increase it if we feel that it is necessary to reduce the amount of matches returned.

* each matching server replies with a list of matching ostfile ids together with the magnitude of error per match and some general usage data (for load balancing).

* the matching redirector replies to the main server (matching results + ident of matching server)

* main server stores the matching data in the db

** (ofid1 int4, ofid2 int4, matching float) as ostfilefoosictb

* command to fetch audio meta data by ostfile id, size+content hash or foosic id

* command to add audio file to mylist by ostfile id or size+content hash (foosic server; UDP)

* same commands as available via TCP (used by the anidb2 cronjob) also available via UDP for use by other clients. i.e. to allow lookups by foosic fingerprint. for that a client would first contact the UDP API (anidb2) with the content hash and if the content hash is unknown to anidb it would send the fingerprint to the foosic server (anidb3) to get one or more foosic id(s) and then use those to query song data from the UDP API. (this is not meant for avdump, but it might be interesting for direct integration into player software, i.e. a winamp/amarok plugin, would work somewhat like the already available musicbrainz plugins)

== General Considerations for Future Expansion ==

* it is very important to effectively limit the number of fingerprints which need to be taken into account for each lookup. As such the file length and the average dom and fit should be stored in a way which allows easy and fast filtering via range queries on those 3 dimensions. so that'd probably mean it will be a: length int4, dom int4, fit int4, fingerprint bytea kind of table

* it may become necessary to purge rarely accessed fingerprints from the db every now and then to limit the db size. in order to do that we'll need to keep some counters and dates. i'd suggest: seencount int4, addeddate timestamp, lastseen timestamp as the foosic server would require no authentication, the same user sending a fingerprint multiple times would increase the counter everytime

** loadbalancing server listens for foosic fingerprint lookups each received lookup is send to _all_ foosic servers

** each foosic server replies with ids or with "unknown"

** loadbalancer merges all id replies together (sorted by error rate) and returns reply to client

** if all foosic servers replied unknown, loadbalancer tells least used server to store the fingerprint and returns the generated id to the client

* this would mean that each query is processed in parallel by all available servers. the very nature of the search approach makes the entire approach very scalable.

** this line is inserted by the matching redirector, the data is only interesting for the main server's cron job. Done only if store=1.

** this line will be suppressed by the matching redirector which processes it to decide where to store a new fingerprint (load balancing)

** this line is inserted by the matching redirector, the data is only interesting for the main server's cron job. Done only if store=1.

** this line will be suppressed by the matching redirector which processes it to decide where to store a new fingerprint (load balancing)

** load factor: a simply multiplicative constant which is used to distinguish between fast and slow server hardware. This can i.e. be used to store twice as many fingerprints on one server compared to others. The fingerprint count is converted according to the following formula prior to comparison/selection of least used server.

*** relative fingerprint number/load = number of fingerprints in db * (load factor / 100)