Avdump: Difference between revisions

From AniDB
Jump to navigation Jump to search
No edit summary
 
(102 intermediate revisions by 19 users not shown)
Line 1: Line 1:
{{eyecatch|Note!|<br><br><br><br>Avdump (version one) has been wholly superseded by {{Avdump-current-version}}. As of 2022-01-01 Avdump has been disabled for auto-creqing. Please update to {{Avdump-current-version}}.<br><br><br><br>}}
<br>
----
<br><br><br><br>
{{TOCright}}
{{TOCright}}


==What is it==
==What is it==
'''Avdump''' is a video codec identification utility - identifies which video codec and audio compression method is used on video files. Furthermore it calculates the used bitrates, the length, identifies the number of used audiochannel and more.<br>
'''Avdump''' is a video codec identification utility - identifies which video codec and audio compression method is used on video files. Furthermore it calculates the used bit rates, the length, identifies the number of used audio channels and more.<br>
It is mainly based on [http://mediainfo.sourceforge.net/en mediainfo], but got enhanced and changed to fit anidb's needs by [[User:Epoximator|Epoximator]].<br>
It is mainly based on [http://mediainfo.sourceforge.net/en mediainfo], but got enhanced and changed to fit AniDB's needs by [[User:Epoximator|epoximator]].<br>
'''Avdump''' was created with the intention to provide anidb with a tool to standarize the input. Furthermore it will become part of the upcoming [[AniDB O'Matic|AOM 0.6]] allowing to autocreq all the generated informations.
'''Avdump''' was created with the intention to provide AniDB with a tool to standardize the input for files and be a part of the upcoming [[AniDB O'Matic|AOM 0.6]]. It does however already support auto-creqing by itself. See [[Avdump/Autocreqing|Avdump/Auto-creqing]].


===Supported formats===
===Supported formats===
* Video: MKV, OGM, AVI, DivX, WMV, QuickTime, Real, MPEG-1, MPEG-2, MPEG-4, DVD (VOB)...
* avi, mkv, ogm, mp4, asf/wmv, mpg, rm, mov
::(Codecs: DivX, XviD, MSMPEG4, ASP, H.264, AVC...)
* Audio: OGG, MP3, WAV, RA, AC3, DTS, AAC, M4A, AU, AIFF...
* Subtitles: SRT, SSA, ASS, SAMI...


===Known issues===
===Known issues===
'''Note:''' Avdump is under development. Don't take the validity of its output for granted. Please report errors/issues.
{{eyecatch|Note|Avdump is under development. Don't take the validity of its output for granted. Please report errors/issues.}}
*'''Currently Avdump only supports lower-case letters in user name, if your AniDB user name has capital letters, lower case them only for Avdump usage (as in, it's not needed to change your AniDB user name). If your AniDB user name contains symbols such as an underscore _, then you will actually need to remove that from your AniDB user name for Avdump usage.'''
*Audio channel detection is very simple:
*Audio channel detection is very simple:
**Number of channels might be wrong. (5.1 vs 6.1, 6.1 vs 7.1)
**Number of channels might be wrong. (5.1 vs 6.1, 6.1 vs 7.1)
**No check of actual data in channels. (dummy channels)
**No check of actual data in channels. (dummy channels)
*MP3 VBR check does only detect bitrate switching. (variable frame size)
*No detection of fps for asf/wmv files.
*Missing bits in obscure containers (ie. not mkv/ogm/avi/mp4). Waiting for mediainfo to support this.
*No detection of video anamorphism in mp4.
**No fps for asf/wmv files.
*Only full parsing of avi, mkv and ogm container formats.
**...
*Avdump is only for Windows at the moment. It works via Wine though.
* Avdump is only for Windows at the moment. It works via Wine though.
*[[Avdump issues|More...]]


==Download==
==Download==
Get your copy of avdump [http://folk.ntnu.no/jovetlea/avdump.rar HERE]! Old versions are [http://folk.ntnu.no/jovetlea/avdump-hist/ here].
Latest version (18th March 2008) is 0.34.
Get your copy of Avdump [http://static.anidb.net/client/avdump.rar HERE]!
 
md5sum rar=225f2058812f601876fd5960bed95b74, exe=ee85a21848db15dc8bee3dd4180be953


==Usage==
==Usage==
<b>Avdump</b> is a simple commandline client and as that is to be used in the dosbox:
'''Avdump''' is a simple [[Wikipedia:Command line interface|command line]] client:
:<tt>avdump [-<options> ...] <media file/folder> [<media file/folder> ...]</tt>
 
===Options===
{|align="center" style="border: 1px solid #aaaaaa; border-collapse: collapse; background: #f9f9f9; line-height: 1.0; font-size: smaller; width: 100%"
! style="border: 1px solid #aaaaaa; font-size: 120%; background: #f2f2f2;" | Switch
! style="border: 1px solid #aaaaaa; font-size: 120%; background: #f2f2f2;" | Info
! style="border: 1px solid #aaaaaa; font-size: 120%; background: #f2f2f2;" | Example
|-
|'''ext''':<comma separated list>||Only process files with the specified extensions.||<tt>ext:mkv,avi</tt>
|-
|'''log''':<file path/name>||Mirror the output to the specified file.||<tt>log:avdump.log</tt>
|-
|'''exp''':<file path/name>||Export ed2k-links to the specified file. Very useful when mass adding files.||<tt>exp:ed2k-links.txt</tt>
|-
|'''ac''':<username>:<udp_api_key>||Enable automatic creqing. You should always use this option when dumping anime files!||<tt>-ac:tard:monkey</tt>
|-
|'''port''':<port number>||Use the specified UDP port for communication with the AniDB server (api.anidb.net). Only needed when behind "strict" firewalls.||<tt>port:1337</tt>
|-
|'''done''':<file path/name>||Save processed-file-paths to the specified file and exlude existing entries.||<tt>done:done.txt</tt>
|-
|'''tout''':<timeout_in_seconds>:<number_of_retries>||Set timeout options.||<tt>tout:15:6</tt>
|-
|'''bsize''':<KB_per_block>:<number_of_blocks>||Set size per block (2048 is default) and number of blocks in the circular buffer for hashing. If ''number of blocks'' is less than 2 then a plain single-threaded hasher is used. Might be useful if you got trouble with the default hasher.||<tt>bsize:512:16</tt>
|}
<br/>
{{eyecatch|Note|'''The ''udp_api_key'' is not your user password!!! see [[Avdump#Auto-creqing|Auto-creqing]] .}}
<br/>
{|align="center" style="border: 1px solid #aaaaaa; border-collapse: collapse; background: #f9f9f9; line-height: 1.0; font-size: smaller; width: 80%"
! style="border: 1px solid #aaaaaa; font-size: 120%; background: #f2f2f2;" | Switch
! style="border: 1px solid #aaaaaa; font-size: 120%; background: #f2f2f2;" | Type
! style="border: 1px solid #aaaaaa; font-size: 120%; background: #f2f2f2;" | Description
|-
|s||Formatting||Use '''s'''hort/simple format.
|-
|l||Formatting||'''L'''ist codecs.
|-
|x||Formatting||Old '''X'''ML format (intended for AOM).
|-
|y||Formatting||New XML format (used when auto-creqing).
|-
|c||Control||Do not re'''c'''urse into sub-folders, only handle files in current folder.
|-
|i||Control||Sk'''i'''p full parsing (only for testing, not compatible with auto-creqing).
|-
|m||Control||'''M'''onitor folder(s). Check for new files every now and then.
|-
|n||Control||Add an extra '''n'''ewline after each file file has been processed.
|-
|p||Control||'''P'''ause when done, keep command window open.
|-
|q||Control||Pause after each file.
|-
|r||Control||Randomize the file order.
|-
|t||Control||Print '''t'''ime used per file.
|-
|z||Control||Delete files after parsing.
|-
|o||Control||Wait for resp'''o'''nse when sending dumps and resend if needed.
|-
|0||Hash||crc32
|-
|1||Hash||ed2k
|-
|2||Hash||md5
|-
|3||Hash||sha1
|-
|5||Hash||tth
|-
|6||Hash||aich
|-
|9||Hash||Mode for hashing music files. Should be used together with <tt>-ac</tt>.
|-
|a||Hash||Include '''a'''ll hash algorithms.
|-
|h||Hash||'''H'''ash only mode, i.e. no metadata parsing.
|-
|e||Hash||Print '''e'''d2k link
|-
|d||Hash||Print Ani'''D'''B link
|-
|g||Hash||Print AniDB link and open it with the default browser.
|-
|u||Hash||Print the time '''u'''sed for hashing per file.
|}
 
===Windows Shortcut===
#Extract Avdump to a folder of choice.
#:Example: <tt>"C:\Program Files\avdump\"</tt>
#Right-click <tt>avdump.exe</tt>, select ''Create Shortcut''.
#Right-click the shortcut, add options to the ''Shortcut - Target'' line.
#:Example, normal usage: <tt>"C:\Program Files\avdump\avdump.exe" -p</tt>
#:Example, auto-creq usage: <tt>"C:\Program Files\avdump\avdump.exe" -o -ac:''myName'':''udp_api_key''</tt>
#::You can also set other options like ''s'', ''x'' or ''t'' if you wish to do so.
#::A commandline option to export ED2K links to a file is by adding -exp: <tt>''C:\Path\filename.txt''</tt>, etc.
#::Note that you need to set an '''UDP API Key''' in your AniDB profile for auto-creqing.
#:Optional: Change ''Run'' to ''Maximized''.
#Drag and drop any media-file onto the shortcut.
 
{{eyecatch|Note|'''The ''udp_api_key'' is not your user password!!! see [[Avdump#Auto-creqing|Auto-creqing]] .}}
 
====Send to====
#Move shortcut to the SendTo folder.
#:Example, under XP: <tt>"C:\Documents and Settings\''Username''\SendTo"</tt>
#:Example, under Vista: <tt>"C:\Users\''Username''\AppData\Roaming\Microsoft\Windows\SendTo"</tt>
#::Where ''Username'' is your Windows Login-Name.
#:Note that ''SendTo'' is a hidden folder.
#Right-click any media-file, go to "Send To", choose <tt>avdump.exe</tt>
#Get [[Creq|CReqing]]
 
===MS-DOS Batch File===
#Extract Avdump to a folder of choice.
#Create a MS-DOS batch file with the 2 lines below, and save it as ''filename.bat''.
#<tt>@ECHO OFF</tt>
#<tt>C:\AVDumpPath\avdump.exe %1 -exp:''C:\Path\filename.txt'' -ac:''myName'':''udp_api_key''</tt>
#Drag and drop any media-file onto the shortcut.
 
{{eyecatch|Note|'''The ''udp_api_key'' is not your user password!!! see [[Avdump#Auto-creqing|Auto-creqing]] .}}


<tt>avdump [-ext:<extensions>] [-log:<log file>] <media file/folder> [<media file/folder> ...]</tt>
===Graphical user interface===
<pre>Options: (one letter switches can be put in one string)
* ''See:'' '''[[Avdump GUI]]'''
Input:
  ext      comma separated extension list
  log      write output to file
  ac      autocreq: '-ac:username:apipass'
  port    udp port used by ac
  exp      export ed2k-links to file
Output: (exclusive)
  s        Short (creq friendly)
  l        List (codecs)
  x        XML Old format
  y        XML Creq format
Control:
  r        Random file order
  p        Pause when done (hold cmd window)
  q        pause after each file
  t        print Time used for each file
  n        add extra Newline after each file
  i        skip full parsing (only for testing)
Hash:
  0        crc32
  1        ed2k
  2        md5
  3        sha1
  5        tth
  6        aich
  a        All hash algorithms
  h        Hash mode (no a/v parsing)
  e        print Ed2k link
  d        print anidb link
  g        print and Goto anidb link
  u        print time used</pre>


===Alternative way of usage===
===Processing DVD's===
*Extract to a folder of choice
For processing DVD's it's best to set a small blocksize. Helps to improve performance.
*Right-click avdump.exe, select "Create Shortcut"
*Right-click the shortcut, add " -p" to the "Shortcut - Target" line
::Example: <tt>"C:\program files\Codecs\avdump\avdump.exe" -p</tt>
::Optional: Change "Run" to "Maximized"
:: you can also set other options like <i>s</i>, ''x'' or ''t'' if you wish to do so.
*Move shortcut to C:\Documents and Settings\[Username]\SendTo
::[Username] is your WinXP LogIn-Name
*Right-click any media-file, goto "Send To", choose avdump (or whatever you called the shortcut)
*Be happy :D
*Get [[Creq|CReqing]]


<i>(thanks to taxit for this tip)</i>
Something like <tt>-bsize:256:1</tt> ''(default is 2048)''


== Autocreqing ==
== Auto-creqing ==
[[Image:Autocreq.gif|thumb|simple diagramm of the way avdump works]]
[[Image:Autocreq.gif|thumb|Simple diagram of the way auto-creqing with Avdump works]]


Avdump is now part of a new autocreqing system. Some more or less important notes:
Avdump is now part of a new [[auto-creqing]] system. Some more or less important notes:
* To be able to use this feature you’ll need an AniDB account and you have to define the ''api password'' in your [[profile]]. All data sent to the server will be logged with ip and uid.
* To be able to use this feature you’ll need an AniDB account '''and you have to define the ''UDP API Key'' in your [[profile]]'''. All data sent to the server will be logged with IP and uid.
* Communication between client and server is ''one-way'' (except session init/refresh); the client does not receive any confirmation for sent data. This means that there is no guaranty that all the dumps you send will arrive at the server and stored in the database, and there is no way to check this.
* Communication between client and server is ''one-way'' (except session init/refresh); the client does not receive any confirmation for sent data. This means that there is no guaranty that all the dumps you send will arrive at the server and stored in the database, and there is no way to check this. ''From version 0.31 it's possible to make Avdump wait for confirmation from the server by using the -o switch.''
* Sessions are based on ''ip+port''; check your firewall and router settings to make sure the port is not changed (NAT/whatever). Use <tt>-port:nnnn</tt> to override the default port avdump use.
* It is possible to run more than one instance of Avdump at once, but not more than ten will work OK. At the moment ''only 10 sessions per ip are allowed by the server'', meaning the eleventh instance will terminate the first.
* It is possible to run more than one instance of avdump at once, but not more than three will work ok. At the moment ''only 10 sessions per ip are allowed by the server'', meaning the fourth instance will terminate the first. Also, if you suffer under [http://en.wikipedia.org/wiki/Network_address_translation NAT], running more than one instance will make it worse.
* There is no direct connection between data sent to AniDB and creqs generated. The data received will just be stored for later processing. There is no way to check the current status for a dump. Usually, it should take ''at least 24 hours'' from the moment you dump a file till the data actually changes. If any irregularities occur, or when there is too much data pending, it will take more time.
* There is no direct connection between data sent to AniDB and creqs generated. The data received will just be stored for later processing. There is no way to check the current status for a dump. Usually, it should take ''at least 48 hours'' from the moment you dump a file till the data actually changes. If any irregularities occur, or when there is too much data pending, it will take more time.
* The creqs generated will report the user who sent the data first (for a specific file) as the creqer.
* The creqs generated will report the user who sent the data first (for a specific file) as the creqer.
* You may dump files currently not in the db. The data is still stored and will be used later if/after the file is added. It might also be used for ''machine assisted file adding'' at some point.
* You may dump files currently not in the database. The data is still stored and will be used later if/after the file has been registered.
* Files creqed by the new system will be locked, meaning some fields will not be possible to change. Notify a moderator if you are '''sure''' that some of the data locked for a specific file is wrong.
* Files creqed by the new system will be locked, meaning some fields will not be possible to change. Notify a moderator if you are '''sure''' that some of the data locked for a specific file is wrong.
* Not all dumps are sent to the server; Only mkv, ogm, avi, mp4, mov, rm and mpeg dumps from non corrupted files are sent:
* Not all dumps are sent to the server; Only mkv, ogm, avi, mp4, mov, rm and mpeg dumps from non corrupted files are sent:
** The asf parser does not detect fps.
** The asf parser does not detect fps.
** Audio files are too few and not considered important atm.
** Audio files are too few and not considered important ATM.
** Nothing useful can be extracted from archives and subtitles (that I can think of).
** Nothing useful can be extracted from archives and subtitles (that I can think of).
** Corruption means that the parser fails and the extracted data can not be trusted.


==Source==
===Why didn't this file get dumped?===
Ask [[User:Epoximator|Epoximator]] for the source if interested.
* It's not avi/mkv/ogm/mp4/mov/mpg/rm.
===Based on===
* The file in AniDB is registered with wrong size and/or ED2K hash.
*MKV : [http://dl.matroska.org/downloads/libebml/ libebml], [http://dl.matroska.org/downloads/libmatroska/ libmatroska], modified [http://svn.matroska.org/viewsvn/trunk/MatroskaUtils/ MatroskaUtils]
* The package never reached the server.
*OGM : [http://www.xiph.org/downloads/ libogg], [http://www.xiph.org/downloads/ libvorbis], modified [http://svn.corecodec.org/ogmtools ogmtools]
* The decryption or decompression failed at server side (rare).
*MP4 : [http://mpeg4ip.cvs.sourceforge.net/mpeg4ip mpeg4ip], [http://mediainfo.cvs.sourceforge.net/mediainfo/ mediainfo]
* The dump is not valid XML (rare).
*AVI : modified [http://mediainfo.cvs.sourceforge.net/mediainfo/ mediainfo]
 
*Everything else : [http://mediainfo.cvs.sourceforge.net/mediainfo/ mediainfo]
===Why isn't this file verified?===
Even if the file is dumped it doesn't mean it will get auto-creqed (and verified). Here are the reasons:
* The file is corrupt/invalid ([http://anidb.net/perl-bin/animedb.pl?show=avmf&do=xml&avid=76226 example]).
* The provided data is considered "incoherent". (Indicates a bug in Avdump.)
* The dump was marked unfit for file verification by an AniDB Moderator.
* The file type is not included in auto-creqing yet (mov/mpg/rm ATM).
 
===What's the current status?===
Look [[Avdump status|here]].
 
==Development==
The source can be found [http://dev.anidb.net/websvn/listing.php?repname=AniDB+CSS&path=%2Ftrunk%2Fudp_clients%2Favdump%2F&rev=0&sc=0 here]. Avdump has only been built on MS Windows systems.
 
===Required libraries===
* zlib 1.2.3 - http://www.zlib.net/ (used by several other libs)
* libebml 0.7.7 - http://dl.matroska.org/downloads/libebml/ (used by MatroskaUtils)
* libmatroska 0.8.1 - http://dl.matroska.org/downloads/libmatroska/ (used by MatroskaUtils)
* libogg 1.1.3 - http://www.xiph.org/downloads/ (used by ogmtools)
* libvorbis 1.1.2 - http://www.xiph.org/downloads/ (used by ogmtools)
* ZenLib CVS (09.09.2007) - http://sourceforge.net/projects/zenlib/ (used by MediaInfoLib)
* MediaInfoLib CVS (09.09.2007)  - http://sourceforge.net/projects/mediainfo/ (avi, mp4, etc. parsing)
* ogmtools SVN (13.08.2005) - http://svn.corecodec.org/ogmtools (ogm parsing)
* MatroskaUtils SVN (26.07.2005) - http://svn.matroska.org/viewsvn/trunk/MatroskaUtils/ (mkv parsing)
* libFooID 1.0 - http://foosic.org/libfooid.php (fooid fingerprint)
* mpg123-0.65 - http://www.mpg123.de/ (mp3 decoding)
* tinyxml 1.0.1 - http://www.grinninglizard.com/tinyxml/ (XML generation)
* wxWidgets-2.8.7 - http://www.wxwidgets.org/ (used by MediaInfoLib and avdump.exe)


==Changelog==
==Changelog==
0.29 : 070113 - importance: high
<div style="height: 280px; overflow: auto; padding: 3px; border:1px solid #AAAAAA; font-size: smaller">
  - Autocreq
0.34.00 : 080316 - no need to redump your collection
  - Ed2k export to file (-exp:<file>).
*New switch 'N'; No data output. Does not affect error messages or progress.
  - Various forgotten fixes and changes.
*Print average hashing speed on -u.
0.28 : 061123 - importance: test version
*Fixed minor issue with avg and std fps for mkv files with first timecode != 0.
  - Fixed broken duration for vorbis in ogm (was 0).
*Executable self-check to prevent bad versions/hw from submitting data.
  - Fixed broken language detection.
 
  - Fixed/removed default track flag in ogm.
0.33.00 : 080106 - importance: medium
  - Changes in the switches: y = XML2+autocreq, c = autocreq.
*Fix: support for SimpleBlock @ mkv (affects only a few files atm).
  - Added SBR detection for AAC in mp4.
*Fix: -done was broken.
  - Added x264 settings string to XML2 for mkv.
*Fix: fps was rounded in -s and default output.
  - New mediainfo.
*New switch '-host:<host name>', default is 'api.anidb.net'.
0.27 : 060927 - importance: low
 
  - WARNING:
0.32.01 : 071212 - importance: low
    * This is an experimental version for a new autocreq system.
*New switch 'v': use \n instead of \r @ progress.
    * Due major internal changes it's likely to be borked.
*New switch 'w': supress progress completely.
    * Only upgrade if you want to help testing; use -c.
*New switch 'ms:<ms>': monitor sleep duration in micro seconds.
    * No actual creq will be filed, but data are gathered.
*Fix: don't export ed2k link on error.
  - Improved/fixed ogm and avi track size calculation.
*Fix: only enumerate writable files.
    - Error was less than ~30 KB for a normal file.
 
  - Fixed samplerate=0 when audio not vorbis and added PCM recognition in ogm parser.
0.32.00 : 071016 - importance: high
  - Fixed wrong AR printed when > 1.70.
*OstFile proto.
  - Fixed missing zeros at beginning in crc32 checksums.
*Fixed 1000000 fps for some mkv files.
0.26 : 060912 - importance: low
*Fixed bitrate for some avi files.
  - Changes in the xml mode; formatted time and exact channels value.
*Fixed hashing for large files.
  - Added option to skip parsing altogether (hash only mode).
*Fixed parsing of large mkv files.
  - Added switch '-ext:{comma separated file extension list}'.
*Fixed .sub files identified as mpg.
  - Changes in the arguments. See usage.
*Faster hashing.
0.25 : 060907
*Both ed2k hashing algorithms.
  - Added md5 hash.
*Latest MediaInfoLib.
  - Added ed2k link output.
 
  - Changes in the arguments. See usage.
0.31 : 070225 - importance: low
  - Fixed extension check (case).
*Fixed batch abort when parser fails to recognize a file.
0.24 : 060902
*Fixed <lib> for some mpg files (95446).
  - Added 5.1 audio bitrate test; if under 100 kbps then print warning.
*Fixed <lib> for one avi file (199976).
  - Added Matroska attachment support; used when checking overhead.
*Added mp3-ignore-metadata hash (test).
  - Added hashing; crc, ed2k, sha1, tth and aich.
*Updated avmf dump protocol.
0.23 : 060817
0.30 : 070123 - importance: high
  - Changed: log file is now utf8.
*Added option '-done:' for keeping track of already processed files.
  - Fixed video vs audio duration issue with non mkv/ogm files. (fid: 252353)
*Added speed calculation for hashing.
0.22 : 060806
*Added timeout (30 sec) and retry (6 times) for session refresh (-tout).
  - Fixed unicode-in-file-name issue.
*Added option for hashing buffer size.
  - Added VFR test for mp4 files.
*Added monitor (-m) option. Re-check folder(s) every minute.
  - Added switch 'c': pause after each file.
*Added delete (-z) option.
  - Changes in AR rages and removed 'Wrong AR'.
*Added timestamp to -t switch.
  - Minor fixes and changes.
*Added option to not recurs into sub-folders (-c).
0.21 : 060726
*Fixed crash on 222427, 29542, 14303.
  - Fixed codec detection in private frame (user data).
*Fixed 'stream not found' for some avi files.
  - Added switches 'a' (avi), 'm' (mkv), 'o' (ogm), and 'f' (mp4): Only parse these types.
*Fixed tth crash for 0 byte files.
  - Added switch 'l': list mode ({file path/name}\t{track #1 codec}\t...\t{track #n codec}).
*Fixed skipping of files with names starting with dot.
  - Added switch 'i': skip full parsing (mkv/ogm/avi). Only for testing. Do _not_ use when creqing.
0.29 : 070113 - importance: high
0.20 : 060717
*Auto-creq
  - Added return code in xml mode and more warnings in the other modes.
*Ed2k export to file (-exp:<file>).
  - Fixed mp4 parser; audio track sizes.
*Various forgotten fixes and changes.
  - Fixed mkv crash on corrupt files (fid: 216911).
0.28 : 061123 - importance: test version
  - Fixed wrong track duration on mkv files where the first timecode is not 0 (fid: 234960).
*Fixed broken duration for vorbis in ogm (was 0).
0.19 : 060711
*Fixed broken language detection.
  - Fixed rm parser.
*Fixed/removed default track flag in ogm.
  - Fixed mpeg parser, video bitrate.
*Changes in the switches: y = XML2+autocreq, c = autocreq.
  - Minor stuff.
*Added SBR detection for AAC in mp4.
  - Short format does _now_ print video duration.
*Added x264 settings string to XML2 for mkv.
  - Improved the mp3 vbr test. (fid: 240676)
*New mediainfo.
0.18 : 060624
0.27 : 060927 - importance: low
  - MP4 fixes: sub tracks, track lang, missing chan/srate.
*WARNING:
  - Updated MediaInfoLib (latest cvs).
:* This is an experimental version for a new auto-creq system.
  - Minor changes in normal output format (duration and size).
:* Due major internal changes it's likely to be borked.
  - Short format does now print video duration (ie. not header value).
:* Only upgrade if you want to help testing; use -c.
0.17 : 060602
:* No actual creq will be filed, but data are gathered.
  - Added extension check.
*Improved/fixed ogm and avi track size calculation.
  - Changes xml output.
:*Error was less than ~30 KB for a normal file.
  - Fixed memory leak when parsing mp4.
*Fixed samplerate=0 when audio not Vorbis and added PCM recognition in ogm parser.
0.16 : 060529
*Fixed wrong AR printed when > 1.70.
  - Fixed some RIFF bugs.
*Fixed missing zeros at beginning in crc32 checksums.
  - Fixed randomizor bug.
0.26 : 060912 - importance: low
  - Updated anidb codec identifiers.
*Changes in the XML mode; formatted time and exact channels value.
  - Added jni.
*Added option to skip parsing altogether (hash only mode).
  - Changes in xml and short mode.
*Added switch '-ext:{comma separated file extension list}'.
0.15 : 060518
*Changes in the arguments. See usage.
  - Updated libebml, libogg, libvorbis.
0.25 : 060907
  - Added switch 'r': random file order.
*Added md5 hash.
  - Added switch 'n': extra newline per file.
*Added ed2k link output.
  - Changes in 'short' output mode.
*Changes in the arguments. See usage.
  - Track numbering starts on 1 now.
*Fixed extension check (case).
  - Changes in ar detection, see: http://wiki.anidb.info/w/Avdump/AR
0.24 : 060902
  - Fixed: Track sizes might be slightly wrong for OpenDML files.
*Added 5.1 audio bitrate test; if under 100 kbps then print warning.
  - Packed exe and dll (UPX 2.00w).
*Added Matroska attachment support; used when checking overhead.
0.14 : 060514
*Added hashing; crc, ed2k, sha1, tth and aich.
  - Added anidb AR, flags and chan str to normal output.
0.23 : 060817
  - Added new output mode: short (-s).
*Changed: log file is now utf8.
  - Added switch -t: show proc time (per file).
*Fixed video vs audio duration issue with non mkv/ogm files. (fid: 252353)
  - Fixed high mem usage bug on some file formats.
0.22 : 060806
  - Fixed access violation on files with very many tracks.
*Fixed unicode-in-file-name issue.
  - Added track names (mkv only).
*Added VFR test for mp4 files.
  - Added version in output.
*Added switch 'c': pause after each file.
  - Some tweaking of output format.
*Changes in AR rages and removed 'Wrong AR'.
0.13 : 060511
*Minor fixes and changes.
  - Fixed wrong track durations for some mkv files.
0.21 : 060726
  - Added anidb channel id (only in xml).
*Fixed codec detection in private frame (user data).
0.12 : 060510
*Added switches 'a' (avi), 'm' (mkv), 'o' (ogm), and 'f' (mp4): Only parse these types.
  - Improved VFR check.
*Added switch 'l': list mode ({file path/name}\t{track #1 codec}\t...\t{track #n codec}).
  - Added anidb video flags (only in xml).
*Added switch 'i': skip full parsing (mkv/ogm/avi). Only for testing. Do _not_ use when creqing.
0.11 : 060509
0.20 : 060717
  - Added anidb codec id for ogm (and fixed a minor issue for some mkv files).
*Added return code in XML mode and more warnings in the other modes.
  - Added simple VFR check for mkv files.
*Fixed mp4 parser; audio track sizes.
0.10 : 060501
*Fixed mkv crash on corrupt files (fid: 216911).
  - Added MP3 VBR check on avi and mkv files (variable frame size check).
*Fixed wrong track duration on mkv files where the first timecode is not 0 (fid: 234960).
  - Added xml output (-x).
0.19 : 060711
  - Added anidb codec id, and some checking on fourcc vs user data.
*Fixed rm parser.
  - Added anidb lang and ar id (only printed in xml mode).
*Fixed mpeg parser, video bitrate.
  - Fixed some issues with mp4, rm and mpg files.
*Minor stuff.
0.09 : 060401
*Short format does _now_ print video duration.
  - Fixed track size for all non mkv/ogm/avi/mp4.
*Improved the mp3 vbr test. (fid: 240676)
  - Changes in the output format.
0.18 : 060624
0.08 : 060330
*MP4 fixes: sub tracks, track lang, missing chan/srate.
  - Exe is now unicode.
*Updated MediaInfoLib (latest cvs).
  - Fixed wrong output for avi files with multiple audio tracks.
*Minor changes in normal output format (duration and size).
  - Added better track size calc for avi files. (first try)
*Short format does now print video duration (i.e. not header value).
  - Added track duration.
0.17 : 060602
0.07 : 060314
*Added extension check.
  - Fixed broken ogm parsing (language).
*Changes XML output.
0.06 : 060305
*Fixed memory leak when parsing mp4.
  - Fixed wrong duration for some mkv files.
0.16 : 060529
  - Fixed wrong track size for some mkv files.
*Fixed some RIFF bugs.
0.05 : 060221
*Fixed randomiser bug.
  - Improved USER data check.
*Updated AniDB codec identifiers.
  - Killed some memory leaks.
*Added jni.
0.04 : 060217
*Changes in XML and short mode.
  - Fixed 'AAC in OGM' crash.
0.15 : 060518
  - Added check for USER data in mkvs and ogms.
*Updated libebml, libogg, libvorbis.
  - Cleaned codec output for mkvs.
*Added switch 'r': random file order.
  - Added some filecomment output.
*Added switch 'n': extra newline per file.
*Changes in 'short' output mode.
*Track numbering starts on 1 now.
*Changes in ar detection, see: http://wiki.anidb.net/w/Avdump/AR
*Fixed: Track sizes might be slightly wrong for OpenDML files.
*Packed exe and dll (UPX 2.00w).
0.14 : 060514
*Added AniDB AR, flags and chan str to normal output.
*Added new output mode: short (-s).
*Added switch -t: show proc time (per file).
*Fixed high mem usage bug on some file formats.
*Fixed access violation on files with very many tracks.
*Added track names (mkv only).
*Added version in output.
*Some tweaking of output format.
0.13 : 060511
*Fixed wrong track durations for some mkv files.
*Added AniDB channel id (only in XML).
0.12 : 060510
*Improved VFR check.
*Added AniDB video flags (only in XML).
0.11 : 060509
*Added AniDB codec id for ogm (and fixed a minor issue for some mkv files).
*Added simple VFR check for mkv files.
0.10 : 060501
*Added MP3 VBR check on avi and mkv files (variable frame size check).
*Added XML output (-x).
*Added AniDB codec id, and some checking on fourcc vs user data.
*Added AniDB lang and ar id (only printed in xml mode).
*Fixed some issues with mp4, rm and mpg files.
0.09 : 060401
*Fixed track size for all non mkv/ogm/avi/mp4.
*Changes in the output format.
0.08 : 060330
*Exe is now Unicode.
*Fixed wrong output for avi files with multiple audio tracks.
*Added better track size calc for avi files. (first try)
*Added track duration.
0.07 : 060314
*Fixed broken ogm parsing (language).
0.06 : 060305
*Fixed wrong duration for some mkv files.
*Fixed wrong track size for some mkv files.
0.05 : 060221
*Improved USER data check.
*Killed some memory leaks.
0.04 : 060217
*Fixed 'AAC in OGM' crash.
*Added check for USER data in mkvs and ogms.
*Cleaned codec output for mkvs.
*Added some filecomment output.
</div>


[[Category:Development]][[Category:Features]][[Category:Avdump]]
[[Category:Features]]
[[Category:AniDB Clients]]
[[Category:Avdump]]
[[Category:Development]]

Latest revision as of 01:05, 1 January 2022

Note!



Avdump (version one) has been wholly superseded by Avdump3. As of 2022-01-01 Avdump has been disabled for auto-creqing. Please update to Avdump3.






What is it

Avdump is a video codec identification utility - identifies which video codec and audio compression method is used on video files. Furthermore it calculates the used bit rates, the length, identifies the number of used audio channels and more.
It is mainly based on mediainfo, but got enhanced and changed to fit AniDB's needs by epoximator.
Avdump was created with the intention to provide AniDB with a tool to standardize the input for files and be a part of the upcoming AOM 0.6. It does however already support auto-creqing by itself. See Avdump/Auto-creqing.

Supported formats

  • avi, mkv, ogm, mp4, asf/wmv, mpg, rm, mov

Known issues

Note Avdump is under development. Don't take the validity of its output for granted. Please report errors/issues.
  • Currently Avdump only supports lower-case letters in user name, if your AniDB user name has capital letters, lower case them only for Avdump usage (as in, it's not needed to change your AniDB user name). If your AniDB user name contains symbols such as an underscore _, then you will actually need to remove that from your AniDB user name for Avdump usage.
  • Audio channel detection is very simple:
    • Number of channels might be wrong. (5.1 vs 6.1, 6.1 vs 7.1)
    • No check of actual data in channels. (dummy channels)
  • No detection of fps for asf/wmv files.
  • No detection of video anamorphism in mp4.
  • Only full parsing of avi, mkv and ogm container formats.
  • Avdump is only for Windows at the moment. It works via Wine though.
  • More...

Download

Latest version (18th March 2008) is 0.34. Get your copy of Avdump HERE!

md5sum rar=225f2058812f601876fd5960bed95b74, exe=ee85a21848db15dc8bee3dd4180be953

Usage

Avdump is a simple command line client:

avdump [-<options> ...] <media file/folder> [<media file/folder> ...]

Options

Switch Info Example
ext:<comma separated list> Only process files with the specified extensions. ext:mkv,avi
log:<file path/name> Mirror the output to the specified file. log:avdump.log
exp:<file path/name> Export ed2k-links to the specified file. Very useful when mass adding files. exp:ed2k-links.txt
ac:<username>:<udp_api_key> Enable automatic creqing. You should always use this option when dumping anime files! -ac:tard:monkey
port:<port number> Use the specified UDP port for communication with the AniDB server (api.anidb.net). Only needed when behind "strict" firewalls. port:1337
done:<file path/name> Save processed-file-paths to the specified file and exlude existing entries. done:done.txt
tout:<timeout_in_seconds>:<number_of_retries> Set timeout options. tout:15:6
bsize:<KB_per_block>:<number_of_blocks> Set size per block (2048 is default) and number of blocks in the circular buffer for hashing. If number of blocks is less than 2 then a plain single-threaded hasher is used. Might be useful if you got trouble with the default hasher. bsize:512:16


Note The udp_api_key is not your user password!!! see Auto-creqing .


Switch Type Description
s Formatting Use short/simple format.
l Formatting List codecs.
x Formatting Old XML format (intended for AOM).
y Formatting New XML format (used when auto-creqing).
c Control Do not recurse into sub-folders, only handle files in current folder.
i Control Skip full parsing (only for testing, not compatible with auto-creqing).
m Control Monitor folder(s). Check for new files every now and then.
n Control Add an extra newline after each file file has been processed.
p Control Pause when done, keep command window open.
q Control Pause after each file.
r Control Randomize the file order.
t Control Print time used per file.
z Control Delete files after parsing.
o Control Wait for response when sending dumps and resend if needed.
0 Hash crc32
1 Hash ed2k
2 Hash md5
3 Hash sha1
5 Hash tth
6 Hash aich
9 Hash Mode for hashing music files. Should be used together with -ac.
a Hash Include all hash algorithms.
h Hash Hash only mode, i.e. no metadata parsing.
e Hash Print ed2k link
d Hash Print AniDB link
g Hash Print AniDB link and open it with the default browser.
u Hash Print the time used for hashing per file.

Windows Shortcut

  1. Extract Avdump to a folder of choice.
    Example: "C:\Program Files\avdump\"
  2. Right-click avdump.exe, select Create Shortcut.
  3. Right-click the shortcut, add options to the Shortcut - Target line.
    Example, normal usage: "C:\Program Files\avdump\avdump.exe" -p
    Example, auto-creq usage: "C:\Program Files\avdump\avdump.exe" -o -ac:myName:udp_api_key
    You can also set other options like s, x or t if you wish to do so.
    A commandline option to export ED2K links to a file is by adding -exp: C:\Path\filename.txt, etc.
    Note that you need to set an UDP API Key in your AniDB profile for auto-creqing.
    Optional: Change Run to Maximized.
  4. Drag and drop any media-file onto the shortcut.
Note The udp_api_key is not your user password!!! see Auto-creqing .

Send to

  1. Move shortcut to the SendTo folder.
    Example, under XP: "C:\Documents and Settings\Username\SendTo"
    Example, under Vista: "C:\Users\Username\AppData\Roaming\Microsoft\Windows\SendTo"
    Where Username is your Windows Login-Name.
    Note that SendTo is a hidden folder.
  2. Right-click any media-file, go to "Send To", choose avdump.exe
  3. Get CReqing

MS-DOS Batch File

  1. Extract Avdump to a folder of choice.
  2. Create a MS-DOS batch file with the 2 lines below, and save it as filename.bat.
  3. @ECHO OFF
  4. C:\AVDumpPath\avdump.exe %1 -exp:C:\Path\filename.txt -ac:myName:udp_api_key
  5. Drag and drop any media-file onto the shortcut.
Note The udp_api_key is not your user password!!! see Auto-creqing .

Graphical user interface

Processing DVD's

For processing DVD's it's best to set a small blocksize. Helps to improve performance.

Something like -bsize:256:1 (default is 2048)

Auto-creqing

Simple diagram of the way auto-creqing with Avdump works

Avdump is now part of a new auto-creqing system. Some more or less important notes:

  • To be able to use this feature you’ll need an AniDB account and you have to define the UDP API Key in your profile. All data sent to the server will be logged with IP and uid.
  • Communication between client and server is one-way (except session init/refresh); the client does not receive any confirmation for sent data. This means that there is no guaranty that all the dumps you send will arrive at the server and stored in the database, and there is no way to check this. From version 0.31 it's possible to make Avdump wait for confirmation from the server by using the -o switch.
  • It is possible to run more than one instance of Avdump at once, but not more than ten will work OK. At the moment only 10 sessions per ip are allowed by the server, meaning the eleventh instance will terminate the first.
  • There is no direct connection between data sent to AniDB and creqs generated. The data received will just be stored for later processing. There is no way to check the current status for a dump. Usually, it should take at least 24 hours from the moment you dump a file till the data actually changes. If any irregularities occur, or when there is too much data pending, it will take more time.
  • The creqs generated will report the user who sent the data first (for a specific file) as the creqer.
  • You may dump files currently not in the database. The data is still stored and will be used later if/after the file has been registered.
  • Files creqed by the new system will be locked, meaning some fields will not be possible to change. Notify a moderator if you are sure that some of the data locked for a specific file is wrong.
  • Not all dumps are sent to the server; Only mkv, ogm, avi, mp4, mov, rm and mpeg dumps from non corrupted files are sent:
    • The asf parser does not detect fps.
    • Audio files are too few and not considered important ATM.
    • Nothing useful can be extracted from archives and subtitles (that I can think of).

Why didn't this file get dumped?

  • It's not avi/mkv/ogm/mp4/mov/mpg/rm.
  • The file in AniDB is registered with wrong size and/or ED2K hash.
  • The package never reached the server.
  • The decryption or decompression failed at server side (rare).
  • The dump is not valid XML (rare).

Why isn't this file verified?

Even if the file is dumped it doesn't mean it will get auto-creqed (and verified). Here are the reasons:

  • The file is corrupt/invalid (example).
  • The provided data is considered "incoherent". (Indicates a bug in Avdump.)
  • The dump was marked unfit for file verification by an AniDB Moderator.
  • The file type is not included in auto-creqing yet (mov/mpg/rm ATM).

What's the current status?

Look here.

Development

The source can be found here. Avdump has only been built on MS Windows systems.

Required libraries

Changelog

0.34.00 : 080316 - no need to redump your collection

  • New switch 'N'; No data output. Does not affect error messages or progress.
  • Print average hashing speed on -u.
  • Fixed minor issue with avg and std fps for mkv files with first timecode != 0.
  • Executable self-check to prevent bad versions/hw from submitting data.

0.33.00 : 080106 - importance: medium

  • Fix: support for SimpleBlock @ mkv (affects only a few files atm).
  • Fix: -done was broken.
  • Fix: fps was rounded in -s and default output.
  • New switch '-host:<host name>', default is 'api.anidb.net'.

0.32.01 : 071212 - importance: low

  • New switch 'v': use \n instead of \r @ progress.
  • New switch 'w': supress progress completely.
  • New switch 'ms:<ms>': monitor sleep duration in micro seconds.
  • Fix: don't export ed2k link on error.
  • Fix: only enumerate writable files.

0.32.00 : 071016 - importance: high

  • OstFile proto.
  • Fixed 1000000 fps for some mkv files.
  • Fixed bitrate for some avi files.
  • Fixed hashing for large files.
  • Fixed parsing of large mkv files.
  • Fixed .sub files identified as mpg.
  • Faster hashing.
  • Both ed2k hashing algorithms.
  • Latest MediaInfoLib.

0.31 : 070225 - importance: low

  • Fixed batch abort when parser fails to recognize a file.
  • Fixed <lib> for some mpg files (95446).
  • Fixed <lib> for one avi file (199976).
  • Added mp3-ignore-metadata hash (test).
  • Updated avmf dump protocol.

0.30 : 070123 - importance: high

  • Added option '-done:' for keeping track of already processed files.
  • Added speed calculation for hashing.
  • Added timeout (30 sec) and retry (6 times) for session refresh (-tout).
  • Added option for hashing buffer size.
  • Added monitor (-m) option. Re-check folder(s) every minute.
  • Added delete (-z) option.
  • Added timestamp to -t switch.
  • Added option to not recurs into sub-folders (-c).
  • Fixed crash on 222427, 29542, 14303.
  • Fixed 'stream not found' for some avi files.
  • Fixed tth crash for 0 byte files.
  • Fixed skipping of files with names starting with dot.

0.29 : 070113 - importance: high

  • Auto-creq
  • Ed2k export to file (-exp:<file>).
  • Various forgotten fixes and changes.

0.28 : 061123 - importance: test version

  • Fixed broken duration for vorbis in ogm (was 0).
  • Fixed broken language detection.
  • Fixed/removed default track flag in ogm.
  • Changes in the switches: y = XML2+autocreq, c = autocreq.
  • Added SBR detection for AAC in mp4.
  • Added x264 settings string to XML2 for mkv.
  • New mediainfo.

0.27 : 060927 - importance: low

  • WARNING:
  • This is an experimental version for a new auto-creq system.
  • Due major internal changes it's likely to be borked.
  • Only upgrade if you want to help testing; use -c.
  • No actual creq will be filed, but data are gathered.
  • Improved/fixed ogm and avi track size calculation.
  • Error was less than ~30 KB for a normal file.
  • Fixed samplerate=0 when audio not Vorbis and added PCM recognition in ogm parser.
  • Fixed wrong AR printed when > 1.70.
  • Fixed missing zeros at beginning in crc32 checksums.

0.26 : 060912 - importance: low

  • Changes in the XML mode; formatted time and exact channels value.
  • Added option to skip parsing altogether (hash only mode).
  • Added switch '-ext:{comma separated file extension list}'.
  • Changes in the arguments. See usage.

0.25 : 060907

  • Added md5 hash.
  • Added ed2k link output.
  • Changes in the arguments. See usage.
  • Fixed extension check (case).

0.24 : 060902

  • Added 5.1 audio bitrate test; if under 100 kbps then print warning.
  • Added Matroska attachment support; used when checking overhead.
  • Added hashing; crc, ed2k, sha1, tth and aich.

0.23 : 060817

  • Changed: log file is now utf8.
  • Fixed video vs audio duration issue with non mkv/ogm files. (fid: 252353)

0.22 : 060806

  • Fixed unicode-in-file-name issue.
  • Added VFR test for mp4 files.
  • Added switch 'c': pause after each file.
  • Changes in AR rages and removed 'Wrong AR'.
  • Minor fixes and changes.

0.21 : 060726

  • Fixed codec detection in private frame (user data).
  • Added switches 'a' (avi), 'm' (mkv), 'o' (ogm), and 'f' (mp4): Only parse these types.
  • Added switch 'l': list mode ({file path/name}\t{track #1 codec}\t...\t{track #n codec}).
  • Added switch 'i': skip full parsing (mkv/ogm/avi). Only for testing. Do _not_ use when creqing.

0.20 : 060717

  • Added return code in XML mode and more warnings in the other modes.
  • Fixed mp4 parser; audio track sizes.
  • Fixed mkv crash on corrupt files (fid: 216911).
  • Fixed wrong track duration on mkv files where the first timecode is not 0 (fid: 234960).

0.19 : 060711

  • Fixed rm parser.
  • Fixed mpeg parser, video bitrate.
  • Minor stuff.
  • Short format does _now_ print video duration.
  • Improved the mp3 vbr test. (fid: 240676)

0.18 : 060624

  • MP4 fixes: sub tracks, track lang, missing chan/srate.
  • Updated MediaInfoLib (latest cvs).
  • Minor changes in normal output format (duration and size).
  • Short format does now print video duration (i.e. not header value).

0.17 : 060602

  • Added extension check.
  • Changes XML output.
  • Fixed memory leak when parsing mp4.

0.16 : 060529

  • Fixed some RIFF bugs.
  • Fixed randomiser bug.
  • Updated AniDB codec identifiers.
  • Added jni.
  • Changes in XML and short mode.

0.15 : 060518

  • Updated libebml, libogg, libvorbis.
  • Added switch 'r': random file order.
  • Added switch 'n': extra newline per file.
  • Changes in 'short' output mode.
  • Track numbering starts on 1 now.
  • Changes in ar detection, see: http://wiki.anidb.net/w/Avdump/AR
  • Fixed: Track sizes might be slightly wrong for OpenDML files.
  • Packed exe and dll (UPX 2.00w).

0.14 : 060514

  • Added AniDB AR, flags and chan str to normal output.
  • Added new output mode: short (-s).
  • Added switch -t: show proc time (per file).
  • Fixed high mem usage bug on some file formats.
  • Fixed access violation on files with very many tracks.
  • Added track names (mkv only).
  • Added version in output.
  • Some tweaking of output format.

0.13 : 060511

  • Fixed wrong track durations for some mkv files.
  • Added AniDB channel id (only in XML).

0.12 : 060510

  • Improved VFR check.
  • Added AniDB video flags (only in XML).

0.11 : 060509

  • Added AniDB codec id for ogm (and fixed a minor issue for some mkv files).
  • Added simple VFR check for mkv files.

0.10 : 060501

  • Added MP3 VBR check on avi and mkv files (variable frame size check).
  • Added XML output (-x).
  • Added AniDB codec id, and some checking on fourcc vs user data.
  • Added AniDB lang and ar id (only printed in xml mode).
  • Fixed some issues with mp4, rm and mpg files.

0.09 : 060401

  • Fixed track size for all non mkv/ogm/avi/mp4.
  • Changes in the output format.

0.08 : 060330

  • Exe is now Unicode.
  • Fixed wrong output for avi files with multiple audio tracks.
  • Added better track size calc for avi files. (first try)
  • Added track duration.

0.07 : 060314

  • Fixed broken ogm parsing (language).

0.06 : 060305

  • Fixed wrong duration for some mkv files.
  • Fixed wrong track size for some mkv files.

0.05 : 060221

  • Improved USER data check.
  • Killed some memory leaks.

0.04 : 060217

  • Fixed 'AAC in OGM' crash.
  • Added check for USER data in mkvs and ogms.
  • Cleaned codec output for mkvs.
  • Added some filecomment output.
Retrieved from "https://wiki.anidb.net/index.php?title=Avdump&oldid=24065"