Avdump3: Difference between revisions

From AniDB
Jump to navigation Jump to search
No edit summary
No edit summary
Line 1: Line 1:
AVDump is a tool to extract meta information from media files while at the same time calculating multiple hashes.  
AVDump is a tool to extract meta information from media files while at the same time calculating multiple hashes. Based on that information reports can be generated in multiple forms. Of particular interest is the ability to send those reports back to AniDB and thereby quickly filling in missing metadata for new files.
Based on that information reports can be generated in multiple forms.  
Of particular interest is the ability to send those reports back to AniDB and thereby quickly filling in missing metadata for new files.


===Supported formats===
== Quickstart ==
Most major file formats are supported to some degree. While hashes are created for all file types, stream details remain shakey for others (notably 'swf'). Because of that for some filetypes (underlined) only the hashes get auto-creqed.


* '''Video files:''' asf/wmv, avi, flv, m2ts, mk3d, mkv, mov, mp4, mpg/mpeg, ogm, ogv, qt, rm/rmvb, <u>swf</u>, ts, webm
Since you’re here you probably just want to sent AVMF Packages (or Dumps) to AniDB which will add metadata to previously (or soon to be) added files. If that doesn’t mean anything to you, you probably want to start with https://wiki.anidb.net/Content:Files and https://wiki.anidb.net/Auto-creqing and https://wiki.anidb.net/Tutorial:How_to_Add_Files_for_Dummies! .
* '''Subtitle files:''' ass, idx, js, lrc, mks, pjs, rt, smi, srt, ssa, sub, sup, tmp, tts, txt, xss
* '''Audio files:''' aac, ac3, dts, dtshd, flac, m4a, mka, mp3, ogg, ra, thd, wav, wma
* '''Archive files:''' 7z, ace, rar, zip
* '''Linker files:''' mkv, smil


===Download===
Anyway lets keep this short:<br />
[http://static.anidb.net/client/avdump3/avdump3_xxxx.zip DOWNLOAD(ZIP)] (Build xxxx) MD5 = xxxxxxxxxxxxxxxxxxxxxxxx
1. Login into AniDB with your account and visit https://anidb.net/user/setting, there goto the Account tab and set a password for “UDP API Key”<br />
2. Make sure Net Core 3.1 or higher is installed<br />
3. Download the latest Version of AVD3 with the link in the section below and extract it somewhere.<br />
4. Start a terminal and navigate to AVDump3<br />
5. Use the following arguments as the bare minimum<br />
5.1. Windows: <code>AVDump3CL.exe --Auth=&lt;YourUserName&gt;:&lt;YourUdpApiPassword&gt; &lt;APathToTheFiles&gt;</code><br />
5.2. Linux: <code>dotnet AVDump3CL.dll --Auth=&lt;YourUserName&gt;:&lt;YourUdpApiPassword&gt; &lt;APathToTheFiles&gt;</code>


===Quickstart===
== Download ==
'''Avdump3''' is a simple [[Wikipedia:Command line interface|command line]] client:
:<tt>AVDump3CL.exe [--<Arg>[=<Value>] -<OneLetterArg> ...] <media file/folder> [<media file/folder> ...]</tt>


===Arguments===
http://static.anidb.net/client/avdump2/avdump3_8134.zip
NameSpace: AniDB


--UploadErrors          |  (False)
== What’s new compared to AVD2? ==
Enables the automatic upload of program errors.


--Authentication, --Auth | --Auth=<username>:<api_key> (<none given>)
AVD3 is a complete rewrite of AVD2 which uses .NET Core instead of .NET Framework and treats Linux as a first class citizen. So it should run on Linux just as well as on Windows. To name a few other big differences:
Enables ACReqing when valid credentials are provided.
Also enables error Reporting! (Add --NoEUp to disable)
Visit anidb.net/perl-bin/animedb.pl?show=profile (Account Tab) to set the api key


--HostEndPoint, --Host  | --Host=<hostname>:<hostport> (api.anidb.info:9002:U)
* Can process multiple files in parallel
Change endpoint of AniDB UDP API server
* Uses native code to speed up hashing significantly
* More efficient reading
* Can move/rename files based on scripts
* More hash algorithms


--LocalPort, --LPort    | --LPort=<localport> (0)
== Arguments ==
Local UDP port used for ACReqing


--Timeout, --TOut        | --TOut=<seconds>:<retries> (20:3)
For more detailed information please run AVD3 with <code>--Help</code>!
Sets the retry count and the timeout before resending the dump


--ACreqErrorPath        |  (<null>)
'''NameSpace''': FileDiscovery '''–Recursive, -R''' Recursively descent into Subdirectories
--Ed2KLogPath            | --Ed2KLogPath=<path> (<null>)
Appends the ED2K-Link after a file has been processed into specified file separated by a line feed character (\n)


'''–ProcessedLogPath, –PLPath''' Appends the full filepath to the specified path


NameSpace: Diagnostics
'''–SkipLogPath, –SLPath''' Filepaths contained in the specified file will not be processed


--SaveErrors            | --SaveErrors (False)
'''–DoneLogPath, –DLPath''' Will set –SkipLogPath and –ProcessedLogPath to the specified filepath
Errors occuring during program execution will be saved to disk.


--SkipEnvironmentElement | --SkipEnvironmentElement (False)
'''–WithExtensions, –WExts''' Only/Don’t Process files with selected Extensions
Skip the environment element in error files


--IncludePersonalData    | --IncludePersonalData (False)
'''–Concurrent, –Conc''' Sets the maximal number of files which will be processed concurrently. First param (max) sets a global limit. (path,max) pairs sets limits per path.
Various places may include personal data. Currently this only affects error files, which will then include the full filepath


--ErrorDirectory        | If --SaveErrors is specified the error files will be placed in the specified path (D:\Projects\C#\AVDump3\AVDump3CL\bin\x64\Debug\netcoreapp3.1)
'''NameSpace''': Processing '''–ProducerMinReadLength'''
--ErrorDirectory=<DirectoryPath>


--NullStreamTest        | --NullStreamTest=<StreamCount>:<StreamLength in MiB>:<ParallelStreamCount> ()
'''–ProducerMaxReadLength'''
Use Memory as the DataSource for HashSpeed testing. Overrides any FileDiscovery Settings!


'''–PrintAvailableSIMDs'''


NameSpace: Display
'''–PauseBeforeExit, –PBExit''' Pause console before exiting


--HideBuffers              | --HideBuffers (False)
'''–BufferLength, –BLength''' Circular buffer size for hashing
Hides buffer bars


--HideFileProgress        | --HideFileProgress (False)
'''–Consumers, –Cons''' Select consumers to use. Use without arguments to list available consumers
Hides file progress


--HideTotalProgress        | --HideTotalProgress (False)
'''NameSpace''': FileMove '''–Test'''
Hides total progress


--ShowDisplayJitter        | --ShowDisplayJitter (False)
'''–LogPath'''
Displays the time taken to calculate progression stats and drawing to console


--ForwardConsoleCursorOnly | --ForwardConsoleCursorOnly (False)
'''–Mode'''
The cursor position of the console will not be explicitly set. This option will disable most progress output.


'''–Pattern''' Available Placeholders ${Name}: FullName, FileName, FileExtension, FileNameWithoutExtension, DirectoryName, SuggestedExtension, Hash-<Name>-&lt;2|4|8|10|16|32|32Hex|32Z|36|62|64&gt;-&lt;OC|UC|LC&gt;


NameSpace: FileDiscovery
'''–DisableFileMove'''


--Recursive, -R              | --Recursive (False)
'''–DisableFileRename'''
Recursively descent into Subdirectories


--ProcessedLogPath, --PLPath | --ProcessedLogPath=<Filepath> (<null>)
'''–Replacements'''
Appends the full filepath to the specified path


--SkipLogPath, --SLPath      | --SkipLogPath=<FilePath> (<null>)
'''NameSpace''': Reporting '''–PrintHashes''' Print calculated hashes in hexadecimal format to console
Filepaths contained in the specified file will not be processed


--DoneLogPath, --DLPath      | --DoneLogPath=<Filepath> (<null>)
'''–PrintReports''' Print generated reports to console
Will set --SkipLogPath and --ProcessedLogPath to the specified filepath


--Concurrent, --Conc        | --Concurrent=<max>[:<path1>,<max1>;<path2>,<max2>;...] (1)
'''–Reports''' Select reports to use. Use without arguments to list available reports
Sets the maximal number of files which will be processed concurrently.
First param (max) sets a global limit. (path,max) pairs sets limits per path.


--WithExtensions, --WExts    | --WithExtensions=[-]<Extension1>[,<Extension2>,...] ()
'''–ReportDirectory, –RDir''' Reports will be saved to the specified directory
Only/Don't Process files with selected Extensions


'''–ReportFileName''' Reports will be saved/appended to the specified filename The following placeholders ${Name} can be used: FileName, FileNameWithoutExtension, FileExtension, ReportName, ReportFileExtension


NameSpace: Processing
'''–ExtensionDifferencePath, –EDPath''' Logs the filepath if the detected extension does not match the actual extension


--BufferLength, --BLength  | --BufferLength=<Size in MiB> (64)
'''–CRC32Error''' Searches the filename for the calculated CRC32 hash. If not present or different a line with the caluclated hash and the full path of the file is appended to the specified path The regex pattern should contain the placeholder ${CRC32} which is replaced by the calculated hash prior matching. Consumer CRC32 will be force enabled!
Circular buffer size for hashing


--ProducerMinReadLength    |  (1)
'''NameSpace''': Diagnostics '''–SaveErrors''' Errors occuring during program execution will be saved to disk
--ProducerMaxReadLength    |  (8)
--Consumers, --Cons        | --Consumers=<ConsumerName1>[,<ConsumerName2>,...] ()
Select consumers to use. Use without arguments to list available consumers


--PrintAvailableSIMDs      |  (False)
'''–SkipEnvironmentElement''' Skip the environment element in error files
--PauseBeforeExit, --PBExit | --PauseBeforeExit (False)
Pause console before exiting


'''–IncludePersonalData''' Various places may include personal data. Currently this only affects error files, which will then include the full filepath


NameSpace: Reporting
'''–PrintDiscoveredFiles'''


--PrintHashes                      | --PrintHashes (False)
'''–ErrorDirectory''' If –SaveErrors is specified the error files will be placed in the specified path
Print calculated hashes in hexadecimal format to console


--PrintReports                      | --PrintReports (False)
'''–NullStreamTest''' Use Memory as the DataSource for HashSpeed testing. Overrides any FileDiscovery Settings!
Print generated reports to console


--Reports                          | --Reports (<null>)
'''NameSpace''': Display '''–HideBuffers''' Hides buffer bars
Select reports to use. Use without arguments to list available reports


--ReportDirectory, --RDir          | --ReportDirectory=<Directory> (D:\Projects\C#\AVDump3\AVDump3CL\bin\x64\Debug\netcoreapp3.1)
'''–HideFileProgress''' Hides file progress
Reports will be saved to the specified directory


--ReportFileName                    | --ReportFileName=<FileName> (<FileName>.<ReportName>.<ReportFileExtension>)
'''–HideTotalProgress''' Hides total progress
Reports will be saved/appended to the specified filename.
The following placeholders can be used: <FileName>, <FileNameWithoutExtension>, <FileExtension>, <ReportName>, ReportFileExtension


--ExtensionDifferencePath, --EDPath | --EDPath=extdiff.txt ()
'''–ShowDisplayJitter''' Displays the time taken to calculate progression stats and drawing to console
Logs the filepath if the detected extension does not match the actual extension


--CRC32Error                        | --CRC32Error=<Filepath>:<RegexPattern> ((, (?i)<CRC32>))
'''–ForwardConsoleCursorOnly''' The cursor position of the console will not be explicitly set. This option will disable most progress output
Searches the filename for the calculated CRC32 hash. If not present or different a line with the caluclated hash and the full path of the file is appended to the specified path.
The regex pattern should contain the placeholder "<CRC32>" which is replaced by the calculated hash prior matching.


===Cross Platform===
'''NameSpace''': AniDBAvmf '''–LocalPort, –LPort''' Local UDP port used for ACReqing
===Unix-like===
===Windows===
===OSX===
===Docker===


===Windows===
'''–ACreqErrorPath'''


====Windows Shortcut====
'''–Authentication, –Auth''' Enables ACReqing when valid credentials are provided Visit https://anidb.net/user/setting (Account Tab) to set the api key
#Extract Avdump to a folder of choice.
#:Example: <tt>"C:\Program Files\AVDump2\"</tt>
#Right-click <tt>AVDump2CL.exe</tt>, select ''Create Shortcut''.
#Right-click the shortcut, add options to the ''Shortcut - Target'' line.
#:Example, normal usage: <tt>"C:\Program Files\AVDump2\AVDump2CL.exe" -p</tt>
#:Example, auto-creq usage: <tt>"C:\Program Files\AVDump2\AVDump2CL.exe" --Auth=''myName'':''udp_api_key''</tt>
#::You can also set other options from the list above if you wish to do so.
#::A commandline option to export ED2K links to a file is by adding <tt>--Exp=''C:\Path\ed2k.txt''</tt>, etc.
#::Note that you need to set an '''UDP API Key''' in your AniDB profile for auto-creqing.
#:Optional: Change ''Run'' to ''Maximized''.
#Drag and drop any media-file onto the shortcut.


{{eyecatch|Note|'''The ''udp_api_key'' is not your user password!!! see [[Avdump#Auto-creqing|Auto-creqing]] .}}
'''–HostEndPoint, –Host''' Change endpoint of AniDB UDP API server AddressFamily: Can be 4=IPv4, 6=IPv6 or U=Unspecified


====Send to====
'''–Timeout, –TOut''' Sets the retry count and the timeout before resending the dump
#Move the shortcut created above to the SendTo folder.
#:Example, under XP: <tt>"C:\Documents and Settings\''Username''\SendTo"</tt>
#:Example, under Vista: <tt>"C:\Users\''Username''\AppData\Roaming\Microsoft\Windows\SendTo"</tt>
#::Where ''Username'' is your Windows Login-Name.
#:Note that ''SendTo'' is a hidden folder.
#Right-click any media-file, go to "Send To", choose <tt>AVDump2CL.exe</tt>
#Get [[Creq|CReqing]]


====MS-DOS Batch File====
'''NameSpace''': AniDBMisc '''–UploadErrors''' Enables the automatic upload of program errors
#Extract Avdump to a folder of choice.
#Create a MS-DOS batch file with the 2 lines below, and save it as ''filename.bat''.
#<tt>@ECHO OFF</tt>
#<tt>C:\Program Files\AVDump\AVDump2CL.exe %* --Exp=''C:\Path\ed2k.txt'' --Auth=''myName'':''udp_api_key''</tt>
#Drag and drop any media-file onto the shortcut.


{{eyecatch|Note|'''The ''udp_api_key'' is not your user password!!! see [[Avdump#Auto-creqing|Auto-creqing]] .}}
'''–Ed2kLogPath'''
 
===Privacy===
====Sample Exception Report====
    <AVD3CLException thrownOn="2020-05-24 15:54:42.5387">
        <Information>
            <EntryAssemblyVersion>3.0.8067.0</EntryAssemblyVersion>
            <LibVersion>3.0.8067.0</LibVersion>
            <Session>177c9119-7ac3-4c78-96cb-680f83ee5c48</Session>
            <Framework>3.1.4</Framework>
            <OSVersion>Microsoft Windows NT 6.2.9200.0</OSVersion>
            <IntPtr.Size>8</IntPtr.Size>
            <Is64BitOperatingSystem>true</Is64BitOperatingSystem>
            <Is64BitProcess>true</Is64BitProcess>
            <ProcessorCount>8</ProcessorCount>
            <UserInteractive>true</UserInteractive>
            <SystemPageSize>4096</SystemPageSize>
            <WorkingSet>254959616</WorkingSet>
        </Information>
        <Message>Unhandled AppDomain wide Exception</Message>
        <Data />
        <Cause>
            <ArgumentException>
                <Message>Waithandle array may not be empty. (Parameter 'waitHandles')</Message>
                <Stacktrace>
                    <Frame>at System.Threading.WaitHandle.WaitMultiple(ReadOnlySpan`1 waitHandles, Boolean waitAll, Int32 millisecondsTimeout)</Frame>
                    <Frame>at System.Threading.WaitHandle.WaitMultiple(WaitHandle[] waitHandles, Boolean waitAll, Int32 millisecondsTimeout)</Frame>
                    <Frame>at System.Threading.WaitHandle.WaitAll(WaitHandle[] waitHandles)</Frame>
                    <Frame>at AVDump3CL.AVD3CLModule.Process(String[] paths) in D:\Projects\C#\AVDump3\AVDump3CL\AVD3CLModule.cs:line 261</Frame>
                    <Frame>at AVDump3CL.Program.Main(String[] args) in D:\Projects\C#\AVDump3\AVDump3CL\Program.cs:line 101</Frame>
                </Stacktrace>
                <Data />
            </ArgumentException>
        </Cause>
        <Stacktrace />
    </AVD3CLException>
 
===Contribution===

Revision as of 22:15, 8 August 2020

AVDump is a tool to extract meta information from media files while at the same time calculating multiple hashes. Based on that information reports can be generated in multiple forms. Of particular interest is the ability to send those reports back to AniDB and thereby quickly filling in missing metadata for new files.

Quickstart

Since you’re here you probably just want to sent AVMF Packages (or Dumps) to AniDB which will add metadata to previously (or soon to be) added files. If that doesn’t mean anything to you, you probably want to start with https://wiki.anidb.net/Content:Files and https://wiki.anidb.net/Auto-creqing and https://wiki.anidb.net/Tutorial:How_to_Add_Files_for_Dummies! .

Anyway lets keep this short:
1. Login into AniDB with your account and visit https://anidb.net/user/setting, there goto the Account tab and set a password for “UDP API Key”
2. Make sure Net Core 3.1 or higher is installed
3. Download the latest Version of AVD3 with the link in the section below and extract it somewhere.
4. Start a terminal and navigate to AVDump3
5. Use the following arguments as the bare minimum
5.1. Windows: AVDump3CL.exe --Auth=<YourUserName>:<YourUdpApiPassword> <APathToTheFiles>
5.2. Linux: dotnet AVDump3CL.dll --Auth=<YourUserName>:<YourUdpApiPassword> <APathToTheFiles>

Download

http://static.anidb.net/client/avdump2/avdump3_8134.zip

What’s new compared to AVD2?

AVD3 is a complete rewrite of AVD2 which uses .NET Core instead of .NET Framework and treats Linux as a first class citizen. So it should run on Linux just as well as on Windows. To name a few other big differences:

  • Can process multiple files in parallel
  • Uses native code to speed up hashing significantly
  • More efficient reading
  • Can move/rename files based on scripts
  • More hash algorithms

Arguments

For more detailed information please run AVD3 with --Help!

NameSpace: FileDiscovery –Recursive, -R Recursively descent into Subdirectories

–ProcessedLogPath, –PLPath Appends the full filepath to the specified path

–SkipLogPath, –SLPath Filepaths contained in the specified file will not be processed

–DoneLogPath, –DLPath Will set –SkipLogPath and –ProcessedLogPath to the specified filepath

–WithExtensions, –WExts Only/Don’t Process files with selected Extensions

–Concurrent, –Conc Sets the maximal number of files which will be processed concurrently. First param (max) sets a global limit. (path,max) pairs sets limits per path.

NameSpace: Processing –ProducerMinReadLength

–ProducerMaxReadLength

–PrintAvailableSIMDs

–PauseBeforeExit, –PBExit Pause console before exiting

–BufferLength, –BLength Circular buffer size for hashing

–Consumers, –Cons Select consumers to use. Use without arguments to list available consumers

NameSpace: FileMove –Test

–LogPath

–Mode

–Pattern Available Placeholders ${Name}: FullName, FileName, FileExtension, FileNameWithoutExtension, DirectoryName, SuggestedExtension, Hash-<Name>-<2|4|8|10|16|32|32Hex|32Z|36|62|64>-<OC|UC|LC>

–DisableFileMove

–DisableFileRename

–Replacements

NameSpace: Reporting –PrintHashes Print calculated hashes in hexadecimal format to console

–PrintReports Print generated reports to console

–Reports Select reports to use. Use without arguments to list available reports

–ReportDirectory, –RDir Reports will be saved to the specified directory

–ReportFileName Reports will be saved/appended to the specified filename The following placeholders ${Name} can be used: FileName, FileNameWithoutExtension, FileExtension, ReportName, ReportFileExtension

–ExtensionDifferencePath, –EDPath Logs the filepath if the detected extension does not match the actual extension

–CRC32Error Searches the filename for the calculated CRC32 hash. If not present or different a line with the caluclated hash and the full path of the file is appended to the specified path The regex pattern should contain the placeholder ${CRC32} which is replaced by the calculated hash prior matching. Consumer CRC32 will be force enabled!

NameSpace: Diagnostics –SaveErrors Errors occuring during program execution will be saved to disk

–SkipEnvironmentElement Skip the environment element in error files

–IncludePersonalData Various places may include personal data. Currently this only affects error files, which will then include the full filepath

–PrintDiscoveredFiles

–ErrorDirectory If –SaveErrors is specified the error files will be placed in the specified path

–NullStreamTest Use Memory as the DataSource for HashSpeed testing. Overrides any FileDiscovery Settings!

NameSpace: Display –HideBuffers Hides buffer bars

–HideFileProgress Hides file progress

–HideTotalProgress Hides total progress

–ShowDisplayJitter Displays the time taken to calculate progression stats and drawing to console

–ForwardConsoleCursorOnly The cursor position of the console will not be explicitly set. This option will disable most progress output

NameSpace: AniDBAvmf –LocalPort, –LPort Local UDP port used for ACReqing

–ACreqErrorPath

–Authentication, –Auth Enables ACReqing when valid credentials are provided Visit https://anidb.net/user/setting (Account Tab) to set the api key

–HostEndPoint, –Host Change endpoint of AniDB UDP API server AddressFamily: Can be 4=IPv4, 6=IPv6 or U=Unspecified

–Timeout, –TOut Sets the retry count and the timeout before resending the dump

NameSpace: AniDBMisc –UploadErrors Enables the automatic upload of program errors

–Ed2kLogPath