AniDB

UDP API DEV: Difference between revisions

Page Discussion

UDP API DEV (view source)

Revision as of 06:07, 18 December 2011

7,504 bytes added ,  18 December 2011
Undo revision 16194 by (lol reading bold at the top of the page is hard, I'll submit a tracker item) Fansubcounter (talk)
(Undo revision 16194 by (lol reading bold at the top of the page is hard, I'll submit a tracker item) Fansubcounter (talk))
 
(91 intermediate revisions by 20 users not shown)
Line 1: Line 1:
{{TOCright}}
{{TOCright}}
'''[[UDP_API_Definition|UDP API]] DEVELOPMENT'''
'''Please use the tracker for all feature requests and bug reports:''' http://tracker.anidb.net
This page is for laying out larger (or radical) additions / changes to the [[UDP API Definition|UDP API]] using the same format and style as the API documentation. This makes it easy to compare different solutions and to copy over changes to the documentation when finally implemented. Tracker items should suffice for minor requests and bug reports.


'''[[UDP_API_Definition|UDP API]] DEVELOPMENT'''
Feel free to edit / comment / whatever.
* add feature requests here
* feel free to edit / comment / whatever


== Data Commands ==
== Data Commands ==
=== [[UDP_API_Definition#ANIME:_Retrieves_data_for_a_specific_anime|ANIME]] ===
=== [[UDP_API_Definition#ANIME:_Retrieve_Anime_Data|ANIME]] ===
'''Possible Replies:'''
 
* XXX ANIME BEST MATCH
* ANIME aid={int4 id}[&acode={int4}[&lang={str}]]
: ...
* FILE fid={int4 id}[&fcode={int4}&acode={int4}[&lang={str}]]
'''Comments:'''<br>
 
I'd like this command to cope with a couple of different encodings when passed a title, and be explicit over whether it finds only one result or does a best-match guess. --[[User:Rar|Rar]] 04:51, 30 Oct 2005 (CET)
Bit 23 of <tt>acode</tt> (other name) will return language (official title) defined by <tt>lang</tt> where <tt>lang</tt> is a list of ISO 639-1 (?) codes (i.e. <tt>de,en</tt>) ordered by preference. The first existing language will be returned. Applies to 24 (short name list) and 25 (synonym list) also.
: On second thought, is a wide search really wanted or is 'WHERE lower(name)=' good enough?. About encodings, do you care to elaborate? --[[User:Epoximator|Epoximator]] 18:24, 25 November 2005 (CET)
 
::Oh, specifically, I'd like anime lookups to be able to take sjis/utf8 rather than just html escaped ascii, as an option. While all real programming languages have trivial encoding switching: <code>string.decode('sjis').encode('ascii','xmlcharrefreplace')</code> some that might find a use for udp access to anidb (such as the dire mirc scripting language), do not, so dealing with it serverside would be nice. --[[User:Rar|Rar]] 01:28, 29 January 2006 (CET)
An alternative would be to use the language options in the [[profile]].
 
=== [[UDP_API_Definition#EPISODES:_Retrieve_Episodes_Data|EPISODES]] ===
Feature request:
 
* EPISODES aid={int4 id}
return names of all episodes in anime
 
Possible Replies:
* {str eng}|{str romaji}|{str kanji}|...


=== [[UDP_API_Definition#ANIMEINFO:_Retrieves_additional_data_for_a_specific_anime|ANIMEINFO]] ===
: such a command can't be added. The length limit for UDP packets would break it for most anime. And the question is whether a command like this is actually needed in the UDP API. [[User:Exp|Exp]] 07:40, 27 July 2007 (UTC)
'''Syntax:'''
:: well, for now i'm stuck with using wget anime page and than parse it, which is bad, cos there's many unneeded data => more traffic, and changes in html layout breaks compatibility( [[User:iSage|iSage]] 15:26, 27 July 2007 (UTC)
* ANIMEINFO aid={int4 id} | aname={str animename}
::: that's really bad. Why do you need to do that? What are you actually trying to do? Usually you'd use a MyList export as base, import it and only use the UDP API to update the data. [[User:Exp|Exp]] 17:59, 27 July 2007 (UTC)
:::: I need to get a list of English names of all or some episodes of some specified anime. and MyList is not a solution. well, if this is considered as rip off.. sorry then, I'll try to look for another solution  [[User:iSage|iSage]] 12:01, 28 July 2007 (UTC)
::: what for? We might be able to come up with a good solution for your issue, however, we'd need a full account (the big picture!) of what you want to do and why you want to do it. I.e. why do you need those episodes? What triggers the request? What kind of application are we talking about anyway? ... [[User:Exp|Exp]] 20:11, 28 July 2007 (UTC)
:::: web-site with file archive. trigger is: add new anime (or edit)->assign AniDB id->cache episodes. hope you'll understand what i mean. [[User:iSage|iSage]] 06:58, 07 August 2007 (UTC)
::: what's the URL of that page? [[User:Exp|Exp]] 08:28, 7 August 2007 (UTC)
:::: http://aumi.ru --[[User:ISage|ISage]] 17:33, 9 August 2007 (UTC)


'''Replies:'''
=== [[UDP_API_Definition#GROUP:_Retrieve_Group_Data|GROUP]] ===
* This command should return all missing informations from ANIME, sush as companies, external links (ANN, ...), awards, ...
{{missing}}


=== [[UDP_API_Definition#EPISODE:_Retrieves_data_for_a_specific_episode|EPISODE]] ===
=== [[UDP_API_Definition#GROUP:_Retrieves_data_for_a_specific_group|GROUP]] ===
=== [[UDP_API_Definition#FILE:_Retrieve_File_Data|FILE]] ===
=== [[UDP_API_Definition#FILE:_Retrieve_File_Data|FILE]] ===
'''Comments:'''<br>
*FILE size={int8 size}&ed2k={str ed2khash}[&fcode={int4}&acode={int4}]
Pass a list of fields (limit e.g. 10 entries)
 
and return content. This makes it possible to request new fields in the database without
Is the UDP API supporting other checksum formats? Would it be too much trouble to implement i.e. md5 checksums?
breaking API for older clients.
: Needs database support, meaning exp has to grant it. --[[User:Epoximator|Epoximator]] 09:34, 9 July 2007 (UTC)
On updates, only the list of possible fields
:: Uniqueness of all other hashes besides the ed2k hash is not enforced by AniDB. Meaning their use in a FILE lookup might cause some problems in case multiple files with that hash are in the db. [[User:Exp|Exp]] 09:41, 9 July 2007 (UTC)
need to be extendet. -- --[[User:Dinoex|Dinoex]]
 
: I think it's a good idea. Modified a bit. --[[User:Epoximator|Epoximator]] 13:38, 26 November 2005 (CET)
That's odd. How come AOM supports different hash-methods? (I know this may not be asked here) And shouldn't the md5sum be unique, too? I mean uniqueness IS the reason for using hash... Well I guess we can handle everything through ed2k though. [[User:Ainawing|Ainawing]] 15:49, 9 July 2007 (GMT+1)
: Well, the hashes are likely to be unique. However, this is not enforced by AniDB. Whether two distinct files really have the same hash or whether it is just an input error (i.e. wrong c&p) of the submitting user, the result is the same. You're trying to do a lookup for which you expect exactly one result or nothing at all and you end up with multiple results. And yes, most clients do support other hashing methods and also submit such data to AniDB. However, internally they are all based on ed2k hashes AFAIK. [[User:Exp|Exp]] 08:52, 10 July 2007 (UTC)
:: The real question is: Should we enforce uniqueness? It's not really a problem to implement and shouldn't lead to any problems. We have reports on uniqueness and there are ATM 2x2 files with equal md5, which obviously is wrong (if you look at them). Constraints would just mean that DerIdiot doesn't have to go around and fix such entries from time to time. On the other hand, supporting md5 lookup for clients is hardly important and it'll only have a negative impact on performance (although probably unnoticeable). --[[User:Epoximator|Epoximator]] 12:02, 10 July 2007 (UTC)
::: Well, enforcing uniqueness in the DB brings a slight performance overhead with it, but as file additions are a very seldom event, that wouldn't hurt us. We'd have a couple of extra indices on the file table, which would increase the storage requirements of the DB. Though even that wouldn't be all that much. Supporting MD5 hashes on the UDP API might simplify the writing of very simple UDP clients, as there are easily available MD5 libraries for every programming language out there. For ed2k/md4 libs you might have to search for a bit. However, I think I wouldn't go as far as to enforce uniqueness for all our hashes (i.e. sha1 and tth). But it might be seriously worth considering enforcing MD5 uniqueness, especially if we might some day drop ed2k hashes. MD5 hashes would offer a nice fallback in such a case. [[User:Exp|Exp]] 08:52, 11 July 2007 (UTC)
:::: Agreed. Even better, since hashes are only being used for identification and checksumming, and not authentication, there's no real need for uniqueness other than the 1 in several trillion chance of a collision. The odds get even better with SHA1 or Tiger or RIPEMD. ed2k is not exactly a standard hash, and MD4 is simply not present in some languages, unless you obtain an external library. (For example, I bundle a native Python MD4 with my client, since that library is rare outside of Linux distributions.) [[User:Billessig|Billessig]] 09:07, 26 July 2007 (UTC)
Well, how's it looking? There does not seem to be made any progress on this one. Has this been dropped or simply stalled? :-)  [[User:Ainawing|Ainawing]] 11:24, 19 November 2007 (CET)
 
Feature Request:
* FILE aid={int4 anime id}&epno={int4 episode number}&size={int8 size}[&fcode={int4}&acode={int4}]
Basically retrieve file information by anime, epno and size. In case the ed2k hash doesn't match and the file is thus apparently corrupt (and you're too lazy to figure that out yourself) --[[User:Tesiph|Tesiph]] 10:48, 17 December 2007 (CET)
 
* Retrieve related-episode data with the FILE command, making use of bit 5 (or some other unused bit) of the fcode.  Proposed return is a list of episode IDs separated by commas.  File 275580 is a fair example of why this is useful; right now there's no way to show that this is a dual-episode file via the UDP API. --[[User:Radicand|Radicand]] 14:32, 27 March 2008 (UTC)


== Buddy Commands ==
== MyList Commands ==
Group of commands used to administrate your buddylist.
=== MYLISTADD: Add file to MyList ===
Feature request: It'd be useful if the reply of this command would contain the lid of the added file.


=== BUDDYADD ===
'''Command String:'''
* ADDBUDDY uid={int4 buddy uid}
* ADDBUDDY uname={int4 buddy name}
'''Possible Replies:'''
'''Possible Replies:'''
* XXX BUDDY ADDED
* 210 MYLIST ENTRY ADDED
* XXX NO SUCH USER
: {int4 number of entries added}[|{int4 lid}]
* XXX USER IS ALREADY A BUDDY


=== BUDDYDEL ===
''Replies with 3xx and 4xx codes skipped.''
'''Command String:'''
* DELBUDDY uid={int4 buddy uid}
'''Possible Replies:'''
* XXX BUDDY DELETED
* XXX NO SUCH BUDDY
* XXX YOU GOT NO FRIENDS


=== BUDDYACCEPT ===
'''Notes:'''
'''Command String:'''
* The lid of the added file is written in the reply only in case just one file was added. ''Actually, if many files were added, there lid's would by useless to client because it wouldn't know the correspondence between lid's and files.''
* ACCEPTBUDDY uid=NNN
'''Possible Replies:'''
* XXX BUDDY ACCEPTED
* XXX NO SUCH BUDDY
* XXX BUDDY ALREADY ACCEPTED


=== BUDDYDENY ===
== Notify Commands ==
'''Command String:'''
{{missing}}
* DENYBUDDY uid={int4 buddy uid}
'''Possible Replies:'''
* XXX BUDDY DENIED
* XXX NO SUCH USER
* XXX BUDDY ALREADY DENIED


=== BUDDYLIST ===
== Misc Commands ==
'''Command String:'''
=== ENCRYPT : Encrypt Command ===
* BUDDYLIST startat={int2 start at #}
This is a major modification of the UDP API encryption scheme which will break compatibility with
'''Possible Replies:'''
older clients. However, the current system has some weaknesses and only few clients support encryption so far.
* NNN {int2 start} {int2 end} {int2 total} BUDDY LIST
So this seems acceptable.
: {int4 uid}|{str username}|{int2 state}
: ...
'''Info:'''
* state is a 16bit bit field in integer notation (lowest bit first):
  * bit 1 = this user is in your buddylist
  * bit 2 = this user has accepted you
  * bit 3 = this user is waiting for your approval


=== BUDDYSTATE ===
* AES 128bit, '''CBC mode''', PKCS5 padding
'''Command String:'''
** 128bit MD5 sum (raw, don't convert to hex) of API encryption key (API pass) is used as 128bit key.
* BUDDYSTATE startat={int2 start at #}
** 128bit MD5 sum (raw, don't convert to hex) of {str salt} in the reply of the ENCRYPT command is used as 128bit IV.
'''Possible Replies:'''
* XXX {int2 start} {int2 end} {int2 total} BUDDY STATE
: {int4 uid}|{int2 onlinestate}
: ...


'''Info:'''
* a specific binary preamble is added to all encrypted packets (sent by the client or by the server).
* onlinestate is a 16bit bit field in integer notation (lowest bit first):
** i.e. all encrypted packets start with the three bytes \0\1\0 (hex: 00 01 00).
  * bit 1 = http online (user has issued a HTTP request within the last 10 minutes)
  * bit 2 = udp api online (user is currently connected to the udp api server)
  * example: 0=offline, 1=http online, 2=udp api online, 3=http&udp api online


=== [[UDP_API_Definition#NOTIFY:_Notifications|NOTIFY]] : extend ===
* if encryption is enabled, check every incoming packet for preamble.
'''Command String:'''
** if preamble is present: decrypt and process packet. Reply is also encrypted.
* NOTIFY
*** do not automatically disable encryption if the server fails to decrypt a packet, instead discard the packet. Send an unencrypted error reply "XXX DECRYPTION FAILED - PACKET IGNORED".
** otherwise:
*** if it is an ENCRYPT packet, disable encryption, discard the old key and iv and then process the ENCRYPT packet (thus re-enabling encryption with a new key and iv)
*** for any other type of unencrypted packet, discard the packet. Send an unencrypted error reply "XXX ENCRYPTION ENABLED - UNENCRYPTED PACKET IGNORED"


'''Notification Packet Format:'''
* ENCRYPT command:
* Extension of notification:
** add type=0 to disable encryption (making it possible to disable encryption via "ENCRYPT type=0" once it was enabled) (such a command could be sent either encrypted or unencrypted).
  290 NOTIFICATION
** type=0 does an implicit logout (prevents 3rd parties from disabling server encryption, i.e. to sniff plain text notification data).
  {int4 pending_notifies}|{int4 pending_msgs}|{int4 number_of_online_buddys}
** type!=0 does an implicit logout if (and only if!) the user parameter differs from the currently logged in user for that connection.


That way if we requested a notify command and number_of_online_buddys was more than 0 we could then do a BUDDYLIST to get the status of the users.
any more ideas?


=== [[UDP_API_Definition#PUSH:_UDP_Notification_Registration|PUSH]] : extend ===
General goals of this change are:
'''Command String:'''
* use better supported and more secure CBC mode
* PUSH notify={bool push_file_added_notifications}&msg={bool push_msg_added_notifications}'''[&buddy={bool push_buddy_event_notifications}]'''
* make debugging of encryption problems easier for client developers (plain text error replies, allow disabling of encryption, resending ENCRYPT, preamble)
* make it always clear whether a packet is encrypted or plain text (preamble)
* allow clients to recover from encryption failures without having to wait for the connection to timeout (resending ENCRYPT)
* prevent the server from unintentionally sending unencrypted data to a client, i.e. notifications. (no more dropping of encryption on decryption failure)


'''Notification Packet Format:'''
=== USERLIST : Retrieve MyList Data of a specified user [{{colour|red|rejected}}] ===
* Buddy Event Notify:
Feature Request: The purpose is for a client I want to make, which will essentially be a more user friendly method of viewing and manipulating the view of a user's list. Sort by various categories and so on. I would like to be able to view and browse a specified user's list just as I can with the web interface
  273 {int4 notify_packet_id} NOTIFICATION
  {int4 buddy uid}|{int2 event type}
:* Possible event types:
  0 => LOGIN
  1 => LOGOUT
  2 => ACCEPTED
  3 => ADD (?)


== Misc Commands ==
Probably best implemented to behave just like the MyList commands.
=== ENCRYPT ===
Will cause all future messages from the server, except the first (the reply to the ENCRYPT command itself), to be encrypted (128 bit AES). The client will also have to encrypt all future requests sent to the server. All non-encrypted messages will be discarded by the server. The encryption key is the MD5 hash of a special (new/to be added) password defined in the users profile concatenated with the salt string as given in the reply to the ENCRYPT message. A normal AUTH message is still necessary to authenticate and should follow the ENCRYPT command once the API has acknowledged the encryption.


'''Command String:'''
'''Command String:'''<br>
* ENCRYPT user={str name}[&type={int2 type}]
by lid: (userlist user={str name}&id)
* USERLIST user={str name}&lid={int4 lid}
by fid:
* USERLIST user={str name}&fid={int4 fid}
by size+ed2k hash:
* USERLIST user={str name}&size={int4 size}&ed2k={str ed2khash}
by anime + group + epno
* USERLIST user={str name}&aname={str anime name}&gname={str group name}&epno={int4 episode number}
* USERLIST user={str name}&aname={str anime name}&gid={int4 group id}&epno={int4 episode number}
* USERLIST user={str name}&aid={int4 anime id}&gname={str group name}&epno={int4 episode number}
* USERLIST user={str name}&aid={int4 anime id}&gid={int4 group id}&epno={int4 episode number}


'''Possible Replies:'''
'''Possible Replies:'''
* XXX {str salt} ENCRYPTION ENABLED
* 221 USERLIST
* XXX NO SUCH USER
: {int4 lid}|{int4 fid}|{int4 eid}|{int4 aid}|{int4 gid}|{int4 date}|{int2 state}|{int4 viewdate}|{str storage}|{str source}|{str other}
* XXX ENCRYPTION PASSWORD NOT DEFINED
* 322 MULTIPLE FILES FOUND
* XXX NO SUCH TYPE
: {str anime title}|{int episodes}|{str eps with state unknown}|{str eps with state on hhd}|{str eps with state on cd}|{str eps with state deleted}|{str watched eps}|{str group 1 short name}|{str eps for group 1}|...|{str group N short name}|{str eps for group N}
* 321 NO SUCH ENTRY


'''Info:'''
'''Info:'''
* user is username
* The state field provides information about the location and sharing state of a file in MyList.
* type is type of encryption, 1 => 128 bit AES, none other defined atm
* If files are added after hashing, a client should specify the state as 1 (on hdd) (if the user doesn't explicitly select something else).
* encryption pass must be set in profile settings on the website
* eps is a list of episodes, e.g. "1-12,14-16,T1"
* it is not possible to disable the encryption once enabled while staying logged in
* a logout (the logout message needs to be correctly encrypted) or timeout will disable the encryption
* in order to minimize server load, encryption should be disabled by default and should have to be enabled manually by the user in the configuration options.
* the encryption key is md5(special_api_password_of_user+salt)


'''States:'''
  0 - unknown - state is unknown or the user doesn't want to provide this information
  1 - on hdd - the file is stored on HDD (but is not shared)
  2 - on cd - the file is stored on CD
  3 - deleted - the file has been deleted or is not available for other reasons (i.e. re-encoded)


=== ENCODING ===
'''Example:'''
Sets preferred encoding.
  > USERLIST user=baka&aname=gits sac&s=xxxxx
  < 322 MULTIPLE FILES FOUND
  Koukaku Kidoutai STAND ALONE COMPLEX|26||1-26|1-26,S2-S27|||V-A|S2-S27|LMF|20-26|KAA|1-26|AonE|1-19|Anime-MX|1-3,9-20


'''Command String:'''
'''Comments:'''<br>
* ENCODING name={str encoding name}
It would probably be ok to add this, but note that it would be a hassle to fetch a complete MyList. I mean, you can already do this with your own list, but it takes a lot of time because of the datagram rate limit. (It would take at least 8h to fetch my own list). And that is usage we don't really want @ UDP API. And a command that returns all the data at once will, of course, not be allowed. So, in other words, if you just want to check if a friend has a specific file, or what files per anime (multiple files found), then OK, but ''view and browse a specified user's list just as I can with the web interface'' is not likely to happen/be allowed.--[[User:Epoximator|Epoximator]] 11:29, 4 June 2006 (CEST)
:What I'm really interested in is maintaining local dbs of 3/4 user lists and having a client automatically update them every 2/3 weeks. We use this to coordinate information within our country regarding who has what =p. But what we're interested in is a way to be able to selectively retrieve the entire list by some index. Of course if its slow, then it makes sense to maintain it locally and just look for changes. The other thing I really want is to somehow retrieve with MyList file data (or user list file data), the date at which something was added to the list. (I understand this information exists). The idea here is we can locally query these lists for files added within the last 2/3 months. --[[User:path|path]] 11:53, 25 June 2006 (IST)
::That's really not wanted usage for this API. Have you thought about the possibility to use MyList export? You know you can make you own export template and get all the info you want at once? A command that query all changes (lids) within the last 2/3 months (or even weeks) is not going to be added. --[[User:Epoximator|Epoximator]] 12:36, 25 June 2006 (UTC)


'''Possible Replies:'''
== Other ==
* XXX ENCODING ENABLED
=== Web Service API ===
* XXX ENCODING NOT SUPPORTED
I think most of the UDP API functionality could be exposed as a web service. Not only it would  make client programming simpler but could also solve some NAT/Proxy/Firewall UDP problems. This web service would only provide query capabilities, since callbacks (required for notifications) in web services still pose some problems. --[[User:Jassuncao|Jassuncao]] 08:43, 20 September 2006 (UTC)


'''Info:'''
:Web services are for big idiot corps with rooms of servers and no competent programmers. Spend TCP->HTTP->SOAP overhead to save on personnel. --[[User:Rar|Rar]] 21:49, 23 September 2006 (UTC)
* Supported encodings: http://java.sun.com/j2se/1.5.0/docs/guide/intl/encoding.doc.html
* Default: ASCII.


'''Comments:'''
::Since this is not the right place to start a flame war I will not comment.
: This could be included in the AUTH command or an environment command that includes options for encryption, encoding and compression. (In response to rar's request.) --[[User:Epoximator|Epoximator]] 11:48, 31 January 2006 (CET)
: This should definetly become an additional, optional parameter of the AUTH command. There is really no reason why this should be an extra command. It just means two more UDP packets overhead on each client connection. --[[User:Exp|Exp]] 05:28, 8 February 2006 (CET)
[[Category:Development]]
: I think epoxi was more thinking of this command as setting a session state flag, like a parameter in AUTH would, rather than being sent prior to *every* command, which would of course be silly. I see no reason not to have both. --[[User:Rar|Rar]] 16:21, 10 February 2006 (CET)


=== [[UDP_API_Definition#AUTH:_Authing_to_the_AnimeDB|AUTH]] : extend ===
=== HTTP/XML API===
'''Command String:'''
A complementary [[HTTP API Definition|HTTP/XML API]] has been suggested (and approved). See http://forum.anidb.net/viewtopic.php?f=11&t=6871
* AUTH user={str username}&pass={str password}&protover={int4 apiversion}&client={str clientname}&clientver={int4 clientversion}[&nat=1'''&comp=1&enc={str encoding}&mtu{int4 mtu value}''']


'''Info:'''
[[Category:Development]]
* enc=x will change the encoding used. Supported encodings are [http://java.sun.com/j2se/1.5.0/docs/guide/intl/encoding.doc.html these]. The encoding will change right away (including the response), if it is supported, and be reset on logout/timeout.
[[Category:UDP]]
* comp=1 means that the client supports compression ([http://java.sun.com/j2se/1.5.0/docs/api/java/util/zip/Deflater.html DEFLATE]). The server will compress (instead of truncating) the datagrams when needed if this option is enabled. The first two bytes of compressed datagrams will always be set to zero. (So tags should never start with that.)
[[Category:API]]
* Default [http://en.wikipedia.org/wiki/MTU_(networking) mtu] is 1400. Min/max values?
Retrieved from "https://wiki.anidb.net/Special:MobileDiff/5383...16195"
MediaWiki spam blocked by CleanTalk.
MediaWiki spam blocked by CleanTalk.