Avdump2: Difference between revisions

Latest revision as of 01:24, 19 December 2021

Warning

Avdump2 is deprecated; please use Avdump3.

Important

Avdump2 fully replaces Avdump, which is now deprecated and should not be used.

Important

Avdump GUI is a GUI version of Avdump2, for command line skeptics.

Welcome to the remarkably sparse avdump2 wiki page, lovingly hand created over many minutes + some more.

We gleefully embrace build 7100, yet another public release of avd2. It runs against a new build of MediaInfoLibrary (18.12), contains a number of custom solutions for a number of formats, represents a significant investment of time and energy by Arokh, and is just generally nifty.

There is no general need to reprocess existing files with this build, but we don't mind if you do it anyway!

What is it

Avdump2 is a video/audio codec identification utility - identifies which video codec and audio compression method is used on video files. Furthermore it hashes the file with a set of common hashalgorithms and calculates the used bit rates, the length, identifies the number of used audio channels and more.
It, like its predecessor, is mainly based on mediainfo, but significantly enhanced and changed to fit AniDB's needs by Arokh.

Supported formats

Most major file formats are supported to some degree. While hashes are created for all file types, stream details remain shakey for others (notably 'swf'). Because of that for some filetypes (underlined) only the hashes get auto-creqed.

Video files: asf/wmv, avi, flv, m2ts, mk3d, mkv, mov, mp4, mpg/mpeg, ogm, ogv, qt, rm/rmvb, swf, ts, webm
Subtitle files: ass, idx, js, lrc, mks, pjs, rt, smi, srt, ssa, sub, sup, tmp, tts, txt, xss
Audio files: aac, ac3, dts, dtshd, flac, m4a, mka, mp3, ogg, ra, thd, wav, wma
Archive files: 7z, ace, rar, zip
Linker files: mkv, smil

Download

DOWNLOAD(ZIP) (Build 7100) MD5 = a1539a6139315a3c0f24113bb2d2853b
DOWNLOAD(ZIP) (Build 7101) MD5 = 2ba27f6477e8f933b366da65a6a8fdc2

Usage

Avdump2 is a simple command line client:

AVDump2CL.exe [--<Arg> -<OneLetterArg> ...] <media file/folder> [<media file/folder> ...]

While pre 4991 build arguments can still be used, it is recommended to use the new format shown below. The easiest way to convert them to the new format is to run the program with the old args which then displays a notice and the corresponding args in the new format.

Options

Switch	Type	Description	Default	Example
Auth=<username>:<udp_api_key>	Auto-Creqing	Sends file xml metadata dumps to the server for automatic creqing. You should always use this option when dumping anime files! Also enables error reporting.		--Auth=tard:monkey
Host=<hostname>:<port>	Auto-Creqing	Hostname of AniDB UDP API server	--Host=api.anidb.info:9002
LPort=<port>	Auto-Creqing	Local UDP port used for autocreqing. Only needed when behind "strict" firewalls.		--LPort=12345
TOut=<timeoutInSeconds>:<numOfRetries>	Auto-Creqing	Timeout for sending dumps	--TOut=10:3	--TOut=25:6

ACErr=<file path/name>	Logging	Write file paths of failed dumps to file.		--ACErr=acerr.txt
CRCErr=<file path/name>	Logging	Checks if filename contains calculated crc		--CRCErr=crcerr.txt
ExtDiff=<file path/name>	Logging	Write mismatched extensions and file paths to file.		--ExtDiff=extdiff.txt
FixExt=<file path/name>	Logging	Changes the fileextension to the dectected extension and logs the action.		--FixExt=fixext.txt
Done=<file path/name>	Logging	Skip paths present in log and append paths to newly processed files.		--Done=done.txt
HLog=<file path/name>	Logging	Export hashes to file.		--HLog="$CRC$ $ED2K$":hlog.txt
Log=<file path/name>	Logging	Write output to file.		--Log=log.txt

M	Output	MediaInfo dump		-M
m	Output	MediaInfo XML dump		-m
t	Output	Simple text format		-t
C	Output	XML creq format		-C
c	Output	AVD2 creq dump		-c
h	Output	Hash Dump		-h

BSize=<KB_per_block>:<number_of_blocks>	Control	Buffer size for hashing.		--BSize=512:16
Ext=<comma separated list>	Control	Comma separated extension list. Prepend with "-" to exclude.	Process all supported extensions	--Ext=avi,mkv or --Ext=-zip,-sub or --Ext=*
Mon=<seconds>	Control	Monitor sleep duration in seconds.	--Mon=60	--Mon=120
NoErr	Control	Disable error reporting. You should not disable this as it helps finding and fixing bugs.		--NoErr
R	Control	Do not recurse into subfolders.		-R
p	Control	Pause when done (hold cmd window).		-p
P	Control	Pause after each file.		-P
r	Control	Random file order.		-r
q	Control	Suppress progress (silent).		-q
B	Control	Test pure hashingspeed without io limitations		-B
U	Control	Use UTF8 as console output stream		-U

0	Hashing	For CRC (because the CRC of a 0 byte file is 00000000).		-0
1	Hashing	For SHA1 (because of the 1).		-1
2	Hashing	For ED2K (because of the 2).		-2
3	Hashing	For Tiger (because a tiger in the house minimizes the Three Great Risks -- thieves, fire, and evil spirits).		-3
4	Hashing	For MD4 (because of the 4).		-4
5	Hashing	For MD5 (because of the 5).		-5
6	Hashing	For TTH ... because that's what's left :p		-6
a	Hashing	All (available) hash algorithms.		-a

Exp=<file path/name>	AniDB	Export ed2k-links to the specified file. Very useful when mass adding files.		--Exp=ed2k.txt
PrintEd2kLink	AniDB			--PrintEd2kLink
PrintAniDBLink	AniDB			--PrintAniDBLink
OpenAniDBLink	AniDB			--OpenAniDBLink

Windows Shortcut

Extract Avdump to a folder of choice.
Example: "C:\Program Files\AVDump2\"
Right-click AVDump2CL.exe, select Create Shortcut.
Right-click the shortcut, add options to the Shortcut - Target line.
Example, normal usage: "C:\Program Files\AVDump2\AVDump2CL.exe" -p

Example, auto-creq usage: "C:\Program Files\AVDump2\AVDump2CL.exe" --Auth=myName:udp_api_key
You can also set other options from the list above if you wish to do so.

A commandline option to export ED2K links to a file is by adding --Exp=C:\Path\ed2k.txt, etc.

Note that you need to set an UDP API Key in your AniDB profile for auto-creqing.

Optional: Change Run to Maximized.
Drag and drop any media-file onto the shortcut.

Note	The udp_api_key is not your user password!!! see Auto-creqing .

Send to

Move the shortcut created above to the SendTo folder.
Example, under XP: "C:\Documents and Settings\Username\SendTo"

Example, under Vista: "C:\Users\Username\AppData\Roaming\Microsoft\Windows\SendTo"
Where Username is your Windows Login-Name.

Note that SendTo is a hidden folder.
Right-click any media-file, go to "Send To", choose AVDump2CL.exe
Get CReqing

MS-DOS Batch File

Extract Avdump to a folder of choice.
Create a MS-DOS batch file with the 2 lines below, and save it as filename.bat.
@ECHO OFF
C:\Program Files\AVDump\AVDump2CL.exe %* --Exp=C:\Path\ed2k.txt --Auth=myName:udp_api_key
Drag and drop any media-file onto the shortcut.

Note	The udp_api_key is not your user password!!! see Auto-creqing .

Auto-creqing

Avdump2 is the AniDB auto-creqing system. Some more or less important notes:

To be able to use this feature you’ll need an AniDB account and you have to define the UDP API Key in your profile.
All data sent to the server will be logged with IP and uid.
There is no direct connection between data sent to AniDB and creqs generated. The data received will just be stored for later processing.
There is no way to check the current status for a dump. Usually, it should take at least 24 hours from the moment you dump a file till the data actually changes. If any irregularities occur, or when there is too much data pending, it will take more time.
You may dump files currently not in the database. The data is still stored and will be used later if/after the file has been registered.
The creqs generated will report the user who sent the data first (for a specific file) as the creqer.
It is possible to run more than one instance of Avdump2 at once, up to a maximum of ten. At the moment, only 10 sessions per ip are allowed by the server, meaning the eleventh instance will terminate the first.
Files creqed by the new system will be locked, meaning some fields will not be possible to change. Notify a moderator if you are sure that some of the data locked for a specific file is wrong.

Why didn't this file get dumped?

The file in AniDB is registered with wrong size and/or ED2K hash.
The package never reached the server.
The decryption or decompression failed at server side (rare).
The dump is not valid XML (rare).

Why isn't this file verified?

Even if the file is dumped it doesn't mean it will get auto-creqed (and verified). Here are the reasons:

The file is corrupt/invalid (example).
The provided data is considered "incoherent". (Indicates a bug in Avdump.)
The dump was marked unfit for file verification by an AniDB Moderator.

Avdump2 on Linux

Thanks to Mono, Avdump2 runs natively on Linux as well. However, as most development effort goes into the Windows version and due to the heterogenity of Linux distributions, running Avdump2 on Linux isn’t as straightforward as running it on Windows. All Linux-related usage notes can be found on the Avdump2 on Linux page.

Changes From Avdump Classic

AVD2 now runs on the managed .net platform. This will not affect most, as .net comes pre-installed with current versions of windows. XP or older may have to install the platform manually. While it has been tested on mono, such testing has not been exhaustive. We would be very interested in hearing how it behaves in such an environment.
While most command line parameters remain the same, there are some exceptions.
- The "-o" parameter is no longer available in avd2.
- The "-n" parameter is no longer available in avd2.
- The meaning of the numeric parameters has changed.
In the event of an internal error, AVD2 will submit an error report back to AniDB. The error report contains no user-details (see sample), but is attached to the user name supplied with the -ac parameter. While error reports are only submitted if -ac is specified, they can be disabled by the addition of the -noerr parameter. That said, for the purposes of initial testing, we respectfully request that error reports not be disabled, so bugs can be found and fixed.

Sample Exception Report

<FileExceptions>
 <Information>
  <AVDump2CLVersion>0.1.3734.0</AVDump2CLVersion>
  <RunOnBitness>64</RunOnBitness>
  <OS>Microsoft Windows NT 6.1.7601 Service Pack 1</OS>
 </Information>
 <Exceptions>
 <AVDump2CL.Exceptions.AVD2Exception thrownOn="20110303 05.27.26.9254">
 <Message>Error in DisplayBuffer</Message>
 <System.ArgumentOutOfRangeException>
  <Message>
   Non-negative number required.
   Parameter name: totalWidth
  </Message>
  <StackTrace>
   <Frame>
    at System.String.PadHelper(Int32 totalWidth, Char paddingChar, Boolean isRightPadded)
   </Frame>
   <Frame>at l.a(f A_0, Progress A_1)</Frame>
   <Frame>at l.a(f A_0, Dictionary`2& A_1)</Frame>
  </StackTrace>
 </System.ArgumentOutOfRangeException>
 </AVDump2CL.Exceptions.AVD2Exception>
 </Exceptions>
</FileExceptions>

Known issues

In some cases MediaInfoLib (MIL) used by Avdump2 might report an audio stream to have 0 or 3 channels.
Anamorphic .asf/.wmv and .mpg/.mpeg files might not get detected correctly and will report the PAR instead of the DAR.
Some corrupted mkv files may throw a (Thread Aborted) exception, this will be fixed once the updated version of the mkv parser is introduced

Changelog

1.0.7101.0 : 2020.05.16

Allow compressed dumps of up to 14kb

1.0.7100.0 : 2019.04.07

Don't stop execution on big endian systems
Warn when --ACreqErr is set but not --Auth
Don't report DisplayBuffer, LC_ALL, ThreadCreation failed and console resize exceptions
Removed annoying error message when --Auth is not enabled
Fixed MIL Versionstring
MIL Chapter workaround

1.0.7080.0 : 2019.01.08

Removed thrownOn Attribute from Exception reports
Write ACreqErrors to Console
Filter Exceptions which should be reported
Donelog search is now case insensitive
Fix for Matroskafiles when the tracks sections is at the end of the file
Jaco Subtitle detection is now less trigger happy

1.0.7070.0 : 2019.01.02

MIL Updated to Version 18.12
Removed Client side MIL hash check
Skip logging/acreqing for files with errors (except for matroska files with header data)
EBML Parser (Matroska) crash fixed
Matroska Container Parser Update
Matroska Chapter Parsing
Matroska Parser Crash fixed (Missing Track and other corner cases)
Matroska: extra_size node added (Track overhead)
3d flag
Added Mono Runtime Version to exception metadata
Display "Hashing failed" instead of nothing if hashing fails for whatever reason
Accept /? as parameter (same as --help)
Included MAC MIL library for mostly untested mac support
Added Dumpable extensions: trp
Switch --Version also prints MIL Version
Constrained TTH hash Thread to 4 instead of available corecount
MIL Info shuffling for consistency
Wait longer for MIL to return before hard crashing
Fixed -m Crash
MP4 Lag fixed / MIL Chapters bug workaround
Switch --NoMediaInfoProvider (switch is ignored when acreqing)
Don't eat cmd cursor on soft crash
Log acreq failure when preconditions aren't met
Added CodecFeatures node
Various fileextension provider fixes

0.1.6714.0 : 2014.04.24

Check for invalid usernames (to skip check prepend @)
Less forgiving against corrupted files (i.e. older version may have creqed the files and this one doesn't)
Ogg/Ogm fix for corrupted files
Fixed matroska parser corner-case bug for large files

0.1.6647.0 : 2013.03.29

Another attempt to fix OOM issues
Now any byte array used to read files is reused (=> reduces mem usage jumpiness)
Fixed non default blocksize/blockcount crashes
Fixed stackoverflow crashes on some old mono versions (TraceSource)
Added --Version argument (Prints the AVDump2 version)
Fixed some spelling errors
Added resynching for Matroska files (=> less exceptions for corrupted files)

0.1.6525.0 : 2012.11.??

Reuse datablocks in an attempt to fix OOM issues on 32 bit linux (=> also more stable mem usage on all platforms)
Changed Naming: Password to API Key to avoid confusion (Ommina)

0.1.6489.0 : 2012.10.??

Accept extensionless files when --Ext=* is used
Added disabled client exception
Fixed NullExcpetion bug (Invalid program hashes)

0.1.6452.0 : 2012.09.??+x+y+z

Added notice when the test acreq had an invalid format

0.1.6435.0 : 2012.09.??+x+y

Removed "log" extension from acreqable extensions
Added Framework Node in error reports
Added MIL filesize sanity check

0.1.6361.0 : 2012.09.??+x

Changed BlockConsumer error message to state more clearly if the ACReq was sent or not
UTF8 for Error messages

0.1.6359.0 : 2012.09.??

Moved to git repository
Changes lost in the ether

0.1.6248.0 : 2012.08.28

Shiny icon for the executable

New AniDB Communication, should be faster and more reliable.
- Big file reports no longer fail to be ACReqed.
- Wrong Version and Wrong UserName/Password errors are now separated
- Update notification will be shown if a newer version is available

InfoProvider:
- Trimming all value fields
- Fixes serverside parsing errors

FileExtensionProvider:
- Limit ReadLine length
- Fixes errors which can happen when a binary file is parsed

~~Directory: Added workaround for "too long" filepaths~~
FileAdding: On error, display the path of the causing file/directory

MatroskaParser:
- Less strict with corrupted mkv files.
  (If enough data could be read, the ACReq will be sent despite errors)

FileSource: Fixed nullpointer excpetion

CL:
- Shortcut for DoneLogFileNameOnly is now D instead of R (because it clashed with NoRecurse)
- Added version check and notification of new version if present
- Less jumping for the ETA display

Added extensive tracing (removed in public versions)

Lots of other small things

0.1.5290.0 : 2012.05.12

Fixed FileExtensionParser (hopefully)
TXT Report should be fixed now
Fixed ACReq Send Thread crash
AVD2 no longer pauses when not on Windows
Fixed AVD2 not working when not started from same folder as the executable
Added switch --UseCWD
Other changes I haven't remembered yet

0.1.5028.0 : 2011.09.11

Fixed shift of hashvalues (_Display_ only)
Fixed --Ext with "to remove" extensions
Fixed --Mon/--TOut to actually work
Fixed some bug in the old CLI converter
<no ponies>

0.1.5000.0 : 2011.09.11

all previous builds aside of 4991 have been disabled
<more ponies>

0.1.4964.0 : 2011.09.05

all previous builds have been disabled
Sending ACReqs asynchronous
<insert ponies riding down the rainbow and other stuff that has happened>

0.1.4334.0 : 2011.06.15

all previous builds have been disabled
should work under linux via mono now too

0.1.4260.0 : 2011.05.25

all previous builds have been disabled
fix for rare case where all xml submitted would contain an incorrect hash

0.1.4213.0 : 2011.05.14

all previous builds have been disabled
addition of support and detection of new video file type: .mk3d
addition of support and detection of new audio file type: .dtshd, .thd
addition of support and detection of new subtitle file type: .pjs, .rt, .tmp, .tts, .xss, new type of .sub files
addition of support and detection of new linker file type: .smil
addition of support and detection of .zip files starting with magic bytes "PK00PK"

0.1.3958.0 : 2011.03.15 - no need to redump everything

fixed the bug that caused avdump2 not to retry sending a dump when facing a timeout
video-only mp4 files got wrongly detected as m4v
automatic detection of 32/64bit and use of the correct mediainfolib
test if mediainfolib is accessible
check of mediainfolib version to prevent the usage of untested versions
updated to mediainfolib 0.7.42
added -crcerr parameter to log files where the calculated crc is not found in the filename
added -x parameter to create a "parameterless avdump1"-like output
new switch -H to test pure hashing speed
removed Tiger and MD4 checksums from dumps sent to the AniDB server

0.1.3084.0 : 2011.03.12

initial release