Talk:OstDB DEV Foosic: Difference between revisions

I don't see any reasons to use cleartext/ascii here (?):

::: i don't see why it should be public. we already have a public interface and that is the UDP API. (the MATCH command doesn't make any sense for external use atm because of the ofid argument, btw). it shouldn't make much difference @ performance either. and the clients would still have to communicate with the udp api to get any useful info. adding a new public service will just lead to more confusion and expose a service that's main function is only for internal usage --[[User:Epoximator|Epoximator]] 06:49, 22 May 2007 (UTC)

:::: if we don't make it available as a public service the UDP API would have to take it's place. Some usage scenarios will require a way for clients to find out potential ofids for a file by fingerprint. I.e. a music player plugin which retrieves file metadata (ID3 tags) from anidb (like the musicbrainz plugins do) would first try to identify a file by content hash via the UDP API. But if that fails it would try to get a potential match via the foosic fingerprint.

:::: though you're probably right. It would be a lot simpler for client developers if they could simply send a single lookup query with size, content hash and fingerprint to the UDP API and get either a definitve or a potential match as result instead of having to contact multiple APIs.

* avdump sends data to anidb2 udpapi

** And using the songid instead won't work either. As there won't be a ostfile<->song match at the time when the fingerprints are processed. I.e. we might end up with a group of files which all share similar fingerprints but which are all not yet matched to a song. The fact that tey're all sharing a similar fingerprint is highly relavant for the matching process though. So we can't delay the fingerprint analysis inorder to be able to use the song ids as keys. Which is why I'd suggest that we just create new unique ids for the fingerprints.

:: the work load is not defined by how many fps there are in total, but how many that are selected for matching. the idea is that if a slave come up with no results, or only bad ones, then it should have the new fp because it is "unique" on that server. ie. fps should be distribued so that similar fps are spread as much as possible. but the decision depends on how the selection is done:

::yes, that's true, at least in the beginning. it only depends on how many versions there are of each song and how many of them we have registered. (but one point is that we don't know what's considered a good match atm. it has to be adjusted)

** I do agree that it is probably useful to remember which server a specific fingerprint id is stored on. The easiest/efficient approach for that would be to use a couple of the high bits of the ids to store that info. I.e. we could use the highest 4bits (skipping the sign bit) of a 32bit signed integer, leaving us with an effective 27bit room of ids. I guess we wouldn't reach the 130 million entry mark anytime soon. Neither are we likely to fork of to more than 16 servers. Both numbers could easily be extended later by switching to int8 values and using a larger room in the header for the server selection, i.e. 16bit. That would mean that each server would have it's own unique sequence for fingerprint ids (starting at 0) and we'd simply AND a server specific bitmask to it to obtain the external id, when transmitting the data. Which means that the DB ids on the servers are not unique and will not change if we ever decide to change the splitting between server bits and id bits.

::yes, it's useful for administration. if one slave dies it should be simple to redistribute the "lost" fps between the remaining slaves or to a new one. we should also redistribute fps when new slaves are added in general. i still think that the ids should be created and stored in the main db, but it doesn't really matter. i think it would be safer and more clean, though.