AniDB:AJAX development

Page dedicated to AJAX Development.

Background

Some copy/paste from mail's from sphere.

Mail 1

hi

I've seen the basic stuff of what's going on. I've also tried to tweak the XML specification a little to single out those that are general (to all of anime community) and those that are AniDB specific (I use a namespace idea so that AniDB stuff like ids, ratings etc. are clearly marked).

Also tried throwing in a more general resource tag so that anything can be referenced there. (one example is the anime's main image, any weblinks to external databases, official sites, additional links to images and videos etc.)

As I was reading up stuff, it occurred to me that if we had some ajax-ready XML API, it could be far more useful than just for Ajax. Using XSLT, what was traditionally done on server (transforming data to html) can be pushed to the client. If the data is carefully segmented into largely static ones (like basic anime info that doesn't update a lot) and more dynamic ones (like episodes and files), there is potential gain if inter-server caching can be exploited (since the XSL does not change at all, and the XMLs can be chopped up such that some parts do not need to be updated that often and can be locally/ISP cached).

The only shortcoming is that XSL is a pure declarative language. So to achieve some dynamic formatting (e.g. say if you want every odd row in a table to have different style), may need runtime JavaScript.

There are some quirks though... e.g. I can't see the generated html source any more. when I look at source, I see the XML only. dunno whether that's good or bad ATM. [Rar-You're missing the fact the transform doesn't have to run in the browser. libxslt has py bindings, try that. Or just CTRL+A/View Selection Source in foxy.]

one nice thing about this is that the AJAX parts can be built directly on the XML so one common data format may serve the various clients (raw html, xslt, full Ajax web app (see the javeline demo link I placed in the readme file))

anyway, attached a sample of what I was trying out. it consists of only a XSL and XML file. Just open the XML file in your browser and the XSL will be automatically applied. If you think this is interesting enough to the rest of the mods (regardless of whether it needs to be interested, I just thought that if they did not know yet, at least they should know that this is possible?)

BTW, from the looks of things and the political reason you mentioned for not supporting Ajax, I really don't know where to go start or where to go from here. If the general mood is to stick to what we have, does that mean that you and I are just working towards a closed door?

Mail 2

last version doesn't have the language icons, but I made it lookup a publicly available language file (http://schmidt.devlib.org/data/language-names.xml) for the language description translation. took me a while to get used to declarative approach but this version has the nice lang icons we get from AniDB's default styles.

Mail 3

sorry for sending you three mails in a row. this is the final update for some time to come.

following update: --standardized some of the tags to allow dynamic searching.. e.g. when main title is needed and not found, it will grab any title that's available. If maiun image not found it will take any image.

--more proper title management. now titles can be given in any manner as long as it defines a language and the cdata. the xslt will automatically find all the unique values, collect all the languages and display them in one row per unique official title.

--fixed the odd-even row formatting. now the formatting should be correct regardless of how many rows are in the info.

--episode list (no files yet) added. sorted on type and episode number. Episode title is chosen by <defaultlang>...</defaultlang> in the config part of XML. if the language is not found, it will take any language. If no title is found, it will display "unknown title"

--pasted back in the menu (guest access).

--all the links on page is based on anidb.net, so any relative link still link correctly back to AniDB site

--added another test data (a235.xml eh.. chosen for the large number of official languages)

I think I will take a break until there is feedback. xslt is just a middle ground, but it is probably the basis for future Ajax development, as it totally separates the XML's dependency on formatting. If you are still busy, perhaps I can bring this to the forum by creating an Ajax development thread?

Forum

this is the thread that originated it all.

• [1] (old forum)

Some more updates

I've done just a little more but am still holding onto it. Basically, the language codes are now decoded properly using XMLs from ISO sources. Whatever the language code is, be it 2 letters/3 letters etc., and whatever locale, as long as it follows the xx, xx_XX or xx-XX format, it will get decoded into language (+Locale if available). I found that the XML file I used earlier only contains language, so it cannot decode langcodes like pt_BR : Portuguese (Brazil) properly. Now I linked in codes from ISO-639 (language) and ISO-3166 (country) so theoretically, it can decode anything that have been seen before.

To reach out to all people, I was dreaming up these channels (just for discussion):

The Big Picture

HTML

Needed to support ancient browsers. (including some safari and opera that doesn't do xslt).

• Ups: Simplest to read. No magic there as server does everything.

• Downs: Server does everything, which means that server has to dynamically compose every single page (due to personal settings) which I'd been informed making it difficult to exploit remote http caching.

[Fahrenheit - unfortunately it's true that we have some uncommon browsers connecting that could need html layout, exp can list some examples]

XSLT

• Ups: Server throws out the XML (no html at all), which is also used for Ajax. so theoretically, when we have Ajax, xslt comes for free. the style sheet is static. meaning can be cached etc. The XML file is also largely static (no dynamism due to user settings. It only needs to change when anime data actually changes.) Also allows splitting of data into domain. e.g. the same basic anime XML used by all. files and member only data can be sent in another separate XML to be merged on client's end. XSLT is available on almost all modern browsers.

• Downs: There are still some browsers that don't do xslt. [Rar-Name one?]

[fahrenheit - I think rar tried something in these lines before]

[sphere - this is a link to a simple list of browsers that do support xslt natively, [2]. One thing to add is that the support here I'm talking about is NATIVE support. Even on a browser with no NATIVE support, though it is not possible to open XML natively and have the stylesheet kick it, it is possible to do XSLT within the AJAX environment through JavaScript libraries, by manually loading the XML and XSL and merging them using the provided API.]

AJAX

This is supposedly the real workhorse. Unlike opening a XSLT-transformed XML (which does a one time translation), an AJAX client will be a constant webpage that dynamically updates its parts on receipt of new data. E.g. if the user opens another anime, the Ajax page will just request the required XMLs and update itself. The modern pattern is to implement XSLT/JST into it's components. e.g. say I have a div set to contain anime episodes data. After retrieving the XML, instead of doing the traditional parse by JavaScript and manually add in stuff, the modern way is to declare a template before hand (potentially stored statically as a separate file) and use a general translater engine to merge the XML/XSL. makes client development a breeze, not to mention achieving component reuse (the XSL can be used in other pages too!)

• Ups: Rich web experience. due to partitioning of XML data, potentially a large portion can be more static and cachable. Meaning server only have to serve the differences. E.g. the server may serve the same anime and episode info to all people, but may send a per-user XML which contains settings, preferences and MyList stuff. Lighten server load even further? potentially allow even more user customization like the skinning system (server tells client which skin file you use, which can be set by you), coz now, you can change the entire layout of the page by setting a different xslt/jst file sets??

[fahrenheit - you can change the CSS stylesheet on the profile options]

[sphere - CSS stylesheet can only style the fixed DOM, with xslt/jst, the same XML can generate totally different DOMs (which can be further style by individual CSS). But this remains just an option ATM, since we're still emulating the original anime and it seems good enough that there seems to be little suggestions for some changes that are so radical that CSS alone is insufficient :)]

• Downs: requires JavaScript. heavier on client end (but better client than server, IMO). Have to be made more robust coz there may be higher chances of computer amateurs messing stuff up on their client end (as compared to server-spoon-fed html)

[fahrenheit - for the first part it can be impractical for large anime, for the second, can happen with server-spoon-fed html, we had some greasemonkey scripts before, you can mess with anything that has a dom tree]

[sphere - for the first, I'll like to think for data of AniDB's size, it is still mostly ok as we at most deal with one anime at a time, but this is groundless until we can do some testing]

AJAX ++ (javeline)

This is the future. And it is already here. I've already highlighted the link in my mail to Fahrenheit (www.ajax.org). It is a complete JavaScript-only, cross-browser development than can do most things that your typical desktop app can do.... windows, lists, trees. you name it, it has it. What makes it impressive is the ease with which it can be used. cross-browser means that no need to deal with differences in browsers directly. It really takes a lot off the mind of the developer in tinkering with JavaScript hacks to makes things work and allow more energy to be channelled into the important parts : Design. What's more is that the platform is being developed/maintained continuously, so we get "upgrades" automatically when time passes without having to sink in extra manpower to keep things shiny.

• Ups: everything I've mentioned above. did I mention cross-browser?

• Downs: supposedly even more demanding on browser. But from my tests on a typical 1.8 GHz AMD (single-CPU) using Firefox 2, it seems to be still very light on CPU.

[Fahrenheit - It still seems to be a bit incomplete, looks nice, but i don't know if this could handle large amounts of data. Still the framework concept has already been bashed a couple of times on to my head, including but now limited to moochikit]
[sphere - I'm not sure about load tolerance too. This has to be tested if we ever moved there. What attracts me here isn't just the set of features (for which there are many libraries which can do very sophisticated stuff). Rather, it is the ease and intuitive way whereby a 'template' is declared. And these templates are potentially reusable by other people. e.g. From a single AJAX DOM, I could have one template that generates the basic anime info, another that generates the episode list. These two templates can be used on other pages as well to provide the exact view by linking in the template content which can be stored statically somewhere. The typical page designer just do things like 1. create a div, 2. load an XML and a compatible XSL, 3. call some JavaScript function to render the page. Similar to styles, these template files can be customized and 'shared' between users.]

Multilingual support

[Sphere] Added this as afterthought. I tried fooling around with the basic anime info xslt and managed to add in translation time multilingual support. basically I created a very simple string table in another XML that specifies a language and a list of key-value pairs that denotes a key in some default language (English in this case) and a translated value. This will convert all the static strings like menu items, field names etc. to the language specified in the config section of the XML if it can be found. If not, it will use the key as the original string.

Optionally, to save space by not using a long string as key, each entry in the string table could use the @id attribute instead, and references each string by a number. Personally, I find using strings easier as it makes the original XSL more readable...

e.g. <xsl:call-template name="getPresentString"><xsl:with-param name="key">main</xsl:with-param></xsl:call-template>

instead of

<xsl:call-template name="getPresentString"><xsl:with-param name="id" select="'1'" /></xsl:call-template>

example of XML with string table (each language XML can have more than one string table each corresponding to a different language).

<presentation>

<animeinfo>

<stringtable lang="xx">

<string id="1"><key>main</key>

<value>niam</value></string>

<string><key>search</key>

<value>hcraes</value></string>

<string><key>calander</key>

<value>rednalac</value></string>

</stringtable>

</animeinfo>

</presentation>

Can also potentially derive which language XML to use by filename... e.g. animeinfo.it.xml, animeinfo.fr.xml etc. So when AJAX kicks in, it can try to load the correct language file based on default language, or in the absence of the file, use back English (which are embedded as keys in the XSL, so it does not need a separate language file)

Also, the skeleton of a language file can be automatically generated by parsing an xslt and search for <xsl:call-template name="getPresentString"> and then parse the key, and the unique ones in a dummy language XML in the correct structure. This allows people to use the dummy XML to do translation easier.

I actually use <presentation> as the root of "language" file has it could potentially be expanded to point to a wider range of resources.. e.g. some images representing buttons with rendered text may need different version for different language, so they can potentially be embedded into the more appropriately named localization XML file as well.

Files available here

uploaded them here. 2 sample data (right click and "view source" to see the dummy XML data)

1. [3] [4] These 2 dummy data files has lots of titles defined (Some with same official titles get defined over several XML elements, but xslt collects them and display as one row)

2. [5] This is another dummy data one shows the language file in action. I created my own "reverse-speak" (just reversed all words. LOL) I also defined some general online resources that are enumerable, searchable and filterable by xslt. The "image gallery test" below could be expanded to say only display images of characters, or images of a particular character, or only screenshot images from the film etc.

3. [6] This is the XSL file.

4. [7] This is the language file where I define the fake language

5. [8] [9] These 2 are the publicly available XML with language and country codes used to translate the language text (when you mouse over the country flags beside the official titles)

6. [10] This example illustrates simple multilingual by setting Japanese as default language. Notice that static strings (some are not done yet though) gets changed. Also, episode list automatically tries to list those titles of default language as the first title. Other AniDB code lists like category names etc. are not handled (coz I don't have the complete list ready) but can potentially be done so, such that if entries corresponding to specified default language is available, it can get loaded. (Do pardon any translation mistakes as I am not a native speaker)

Comments, propositions and whatever else

Write comments here.