Avdump3: Difference between revisions
(→Reporting: PrintHashes requires Consumers to be set before it will print anything) |
|||
(72 intermediate revisions by 6 users not shown) | |||
Line 2: | Line 2: | ||
== Quickstart == | == Quickstart == | ||
{{eyecatch|Important|[[Avdump_GUI|Avdump GUI]] is a GUI version of Avdump3, for command line skeptics.}} | |||
Since you’re here you probably just want to send AVMF Packages (or Dumps) to AniDB which will add metadata to previously (or soon to be) added files. If that doesn’t mean anything to you, you probably want to start with | Since you’re here you probably just want to send AVMF Packages (or Dumps) to AniDB which will add metadata to previously (or soon to be) added files. If that doesn’t mean anything to you, you probably want to start with [[Content:Files]] and [[Auto-creqing]] and [[Tutorial:How to Add Files for Dummies]]. | ||
Anyway lets keep this short:<br /> | Anyway lets keep this short:<br /> | ||
# Login into AniDB with your account and visit https://anidb.net/user/setting, there | # Login into AniDB with your account and visit https://anidb.net/user/setting, there go to the Account tab and set a password for “UDP API Key”<br /> | ||
# Make sure [https://dotnet. | # Make sure [https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=6.0.0&arch=x64&rid=win10-x64 .Net 6.0] or higher is installed<br /> | ||
# Download the latest Version of AVD3 with the link in the section below and extract it somewhere.<br /> | # Download the latest Version of AVD3 with the link in the section below and extract it somewhere.<br /> | ||
# Start a terminal and navigate to AVDump3<br /> | # Start a terminal and navigate to AVDump3<br /> | ||
Line 14: | Line 14: | ||
#: '''Linux:''' <code>dotnet AVDump3CL.dll --Auth=<YourUserName>:<YourUdpApiPassword> <APathToTheFiles></code> | #: '''Linux:''' <code>dotnet AVDump3CL.dll --Auth=<YourUserName>:<YourUdpApiPassword> <APathToTheFiles></code> | ||
# Optional: See sections below to improve usage experience | # Optional: See sections below to improve usage experience | ||
## For example, to avoid dumping the same files over and over again, add the following parameter: <code>--DoneLogPath=done.txt</code> | |||
# Optional: Strongly consider adding --UploadErrors to the arguments. For more information see https://wiki.anidb.net/Avdump3#UploadErrors_Argument | |||
== Download == | == Download == | ||
[ | [https://cdn.anidb.net/client/avdump3/avdump3_8293_stable.zip DOWNLOAD (ZIP)] (Build 8293) MD5 = 87ed86a9babc172d638071ff1123fdc7 | ||
== What’s new compared to AVD2? == | == What’s new compared to AVD2? == | ||
AVD3 is a complete rewrite of AVD2 which uses .NET | AVD3 is a complete rewrite of AVD2 which uses .NET 6.0 instead of .NET Framework and treats Linux as a first class citizen. So it should run on Linux just as well as on Windows. To name a few other big differences: | ||
* Can process multiple files in parallel | * Can process multiple files in parallel | ||
Line 27: | Line 29: | ||
* More efficient reading | * More efficient reading | ||
* Can move/rename files based on scripts | * Can move/rename files based on scripts | ||
* Redone command line arguments | |||
* More hash algorithms | * More hash algorithms | ||
* Latest version of MediaInfoLib (MIL) is being used (currently | * Latest version of MediaInfoLib (MIL) is being used (currently 2109) | ||
* Support for .vtt subtitle files | |||
* Support for 32bit has been dropped | * Support for 32bit has been dropped | ||
* Support for Linux aarch64 has been added | |||
* Support for MacOS is not yet available (To add support https://github.com/DvdKhl/AVDump3/blob/master/AVDump3NativeLib/src/AVD3MirrorBuffer.c needs to be implemented, help would be appreciated) | * Support for MacOS is not yet available (To add support https://github.com/DvdKhl/AVDump3/blob/master/AVDump3NativeLib/src/AVD3MirrorBuffer.c needs to be implemented, help would be appreciated) | ||
== Supported formats == | |||
Most major file formats are supported to some degree. While hashes are created for all file types, stream details remain shaky for others (notably 'swf'). Because of that for some filetypes (underlined) only the hashes get auto-creqed. | |||
* '''Video files:''' asf/wmv, avi, flv, m2ts, mk3d, mkv, mov, mp4, mpg/mpeg, ogm, ogv, qt, rm/rmvb, <u>swf</u>, ts, webm | |||
* '''Subtitle files:''' ass, idx, js, lrc, mks, pjs, rt, smi, srt, ssa, sub, sup, tmp, tts, txt, vtt, xss | |||
* '''Audio files:''' aac, ac3, dts, dtshd, flac, m4a, mka, mp3, ogg, ra, thd, wav, wma | |||
* '''Archive files:''' <u>7z</u>, <u>ace</u>, <u>rar</u>, <u>zip</u> | |||
* '''Linker files:''' mkv, smil | |||
== UploadErrors Argument == | |||
When enabled errors while processing or program crashes are uploaded to AniDB which makes it far easier to discover and fix bugs. | |||
Error reports sent look like the following: | |||
<AVD3CLException thrownOn="2020-08-11 18:31:34.2837"> | |||
<Information> | |||
<EntryAssemblyVersion>3.0.8163.0</EntryAssemblyVersion> | |||
<LibVersion>3.0.8163.0</LibVersion> | |||
<Session>395e5893-145d-4a19-8fc9-1ddc0f7feed6</Session> | |||
<Framework>3.1.7</Framework> | |||
<OSVersion>Microsoft Windows NT 6.2.9200.0</OSVersion> | |||
<IntPtr.Size>8</IntPtr.Size> | |||
<Is64BitOperatingSystem>true</Is64BitOperatingSystem> | |||
<Is64BitProcess>true</Is64BitProcess> | |||
<ProcessorCount>40</ProcessorCount> | |||
<UserInteractive>true</UserInteractive> | |||
<SystemPageSize>4096</SystemPageSize> | |||
<WorkingSet>239337472</WorkingSet> | |||
</Information> | |||
<Message>CreatingInfoProviders</Message> | |||
<Data> | |||
<FileName>Hidden(7E940D28B4A58A87C4C19B39EB6EA5E2B11CC7B13762DE44BB7338E752F6676722B227DBA942302F956AF1CE040C57D9295CF53AB33B5DE45FD4FAB6FB462E47)</FileName> | |||
</Data> | |||
<Cause> | |||
<InvalidOperationException> | |||
<Message>MediaInfoLib couldn't open the file</Message> | |||
<Stacktrace> | |||
<Frame>at AVDump3Lib.Information.InfoProvider.MediaInfoLibProvider..ctor(String filePath) in D:\Projects\C#\AVDump3\AVDump3Lib\Information\InfoProvider\MediaInfoLibProvider.cs:line 522</Frame> | |||
<Frame>at AVDump3Lib.Information.AVD3InformationModule.<>c.<.ctor>b__4_4(InfoProviderSetup setup) in D:\Projects\C#\AVDump3\AVDump3Lib\Information\AVD3InformationModule.cs:line 30</Frame> | |||
<Frame>at AVDump3Lib.Information.InfoProvider.InfoProviderFactory.Create(InfoProviderSetup setup) in D:\Projects\C#\AVDump3\AVDump3Lib\Information\InfoProvider\InfoProviderFactory.cs:line 40</Frame> | |||
<Frame>at AVDump3CL.AVD3CLModule.<>c__DisplayClass34_0.<CreateFileMetaInfo>b__0(IInfoProviderFactory x) in D:\Projects\C#\AVDump3\AVDump3CL\AVD3CLModule.cs:line 449</Frame> | |||
<Frame>at System.Linq.Enumerable.SelectIListIterator`2.ToArray()</Frame> | |||
<Frame>at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)</Frame> | |||
<Frame>at AVDump3CL.AVD3CLModule.CreateFileMetaInfo(String filePath, ImmutableArray`1 blockConsumers) in D:\Projects\C#\AVDump3\AVDump3CL\AVD3CLModule.cs:line 449</Frame> | |||
</Stacktrace> | |||
<Data /> | |||
</InvalidOperationException> | |||
</Cause> | |||
<Stacktrace /> | |||
</AVD3CLException> | |||
The value within Hidden() is a hash so it is not possible to get the original data back from that value. When --IncludePersonalData is added those values will be shown in clear text. | |||
'''Attention:''' Please be aware that when --IncludePersonalData is active as well, additional data like the commandline arguments (except for passwords) and path information is sent as well! | |||
== Arguments == | == Arguments == | ||
For more detailed information please run AVD3 with <code>--Help</code>! | For more detailed information please run AVD3 with <code>--Help</code>! | ||
=== Special Arguments === | |||
* FROMFILE | |||
If the first argument is called FROMFILE and the next argument is a file path, every line of that file is interpreted as a single argument. | |||
A line which starts with // is interpreted as a comment and ignored. | |||
See https://wiki.anidb.net/Talk:Avdump3 for an example. | |||
* PRINTARGS | |||
Prints the effective arguments used into the terminal. | |||
=== FileDiscovery === | === FileDiscovery === | ||
Line 96: | Line 164: | ||
| style="white-space: nowrap;" | --ProducerMinReadLength | | style="white-space: nowrap;" | --ProducerMinReadLength | ||
| | | | ||
| | | How much data in MiB the reader has to read each time at minimum | ||
| | | style="white-space: nowrap;" | --ProducerMinReadLength | ||
| style="white-space: nowrap;" | 1 | | style="white-space: nowrap;" | 1 | ||
|- | |- | ||
| style="white-space: nowrap;" | --ProducerMaxReadLength | | style="white-space: nowrap;" | --ProducerMaxReadLength | ||
| | | | ||
| | | How much data in MiB the reader is allowed to read each time at most | ||
| | | style="white-space: nowrap;" | --ProducerMaxReadLength | ||
| style="white-space: nowrap;" | 8 | | style="white-space: nowrap;" | 8 | ||
|- | |- | ||
| style="white-space: nowrap;" | --PrintAvailableSIMDs | | style="white-space: nowrap;" | --PrintAvailableSIMDs | ||
| | | | ||
| | | Print available CPU SIMDs | ||
| | | style="white-space: nowrap;" | --PrintAvailableSIMDs | ||
| style="white-space: nowrap;" | False | | style="white-space: nowrap;" | False | ||
|- | |- | ||
Line 143: | Line 211: | ||
| style="white-space: nowrap;" | --Test | | style="white-space: nowrap;" | --Test | ||
| | | | ||
| | | Test FileMove Settings | ||
| | | style="white-space: nowrap;" | --FileMove.Test | ||
| style="white-space: nowrap;" | False | | style="white-space: nowrap;" | False | ||
|- | |- | ||
| style="white-space: nowrap;" | --LogPath | | style="white-space: nowrap;" | --LogPath | ||
| | | | ||
| | | A line is written for each file that has been moved/renamed. (OldPath => NewPath) | ||
| style="white-space: nowrap;" | --FileMove.LogPath=<FilePath> | | style="white-space: nowrap;" | --FileMove.LogPath=<FilePath> | ||
| | | | ||
Line 155: | Line 223: | ||
| style="white-space: nowrap;" | --Mode | | style="white-space: nowrap;" | --Mode | ||
| | | | ||
| | | Determines how the Pattern Argument is going to be interpreted:<br />Inline: Script is directly entered as the argument<br />File: A path pointing to the script file<br />Placeholder: See example for --Pattern<br />CSharpScript: Script in C#<br />DotNetAssembly: Net Core assembly to be loaded | ||
| <nowiki>--FileMove.Mode=<None|PlaceholderInline|PlaceholderFile|CSharpScriptInline|CSharpScriptFile|DotNetAssembly></nowiki> | | <nowiki>--FileMove.Mode=<None|PlaceholderInline|PlaceholderFile|CSharpScriptInline|CSharpScriptFile|DotNetAssembly></nowiki> | ||
| style="white-space: nowrap;" | None | | style="white-space: nowrap;" | None | ||
Line 161: | Line 229: | ||
| style="white-space: nowrap;" | --Pattern | | style="white-space: nowrap;" | --Pattern | ||
| | | | ||
| Available Placeholders ${Name}:<br &/>FullName, FileName, FileExtension, FileNameWithoutExtension, DirectoryName, SuggestedExtension,<br /><nowiki>Hash-<Name>-<2|4|8|10|16|32|32Hex|32Z|36|62|64>-<OC|UC|LC></nowiki> | | Available Placeholders ${Name}:<br &/>FileSize, FullName, FileName, FileExtension, FileNameWithoutExtension, DirectoryName, SuggestedExtension,<br /><nowiki>Hash-<Name>-<2|4|8|10|16|32|32Hex|32Z|36|62|64>-<OC|UC|LC></nowiki> | ||
| style="white-space: nowrap;" | --FileMove.Pattern=${DirectoryName}\${FileNameWithoutExtension}${SuggestedExtension} | | style="white-space: nowrap;" | --FileMove.Pattern=${DirectoryName}\${FileNameWithoutExtension}${SuggestedExtension} | ||
| style="white-space: nowrap;" | ${DirectoryName}\${FileNameWithoutExtension}${FileExtension} | | style="white-space: nowrap;" | ${DirectoryName}\${FileNameWithoutExtension}${FileExtension} | ||
Line 167: | Line 235: | ||
| style="white-space: nowrap;" | --DisableFileMove | | style="white-space: nowrap;" | --DisableFileMove | ||
| | | | ||
| | | Don't move the file even if the Pattern says so | ||
| style="white-space: nowrap;" | --FileMove.DisableFileMove | | style="white-space: nowrap;" | --FileMove.DisableFileMove | ||
| style="white-space: nowrap;" | False | | style="white-space: nowrap;" | False | ||
Line 173: | Line 241: | ||
| style="white-space: nowrap;" | --DisableFileRename | | style="white-space: nowrap;" | --DisableFileRename | ||
| | | | ||
| | | Don't rename the file even if the Pattern says so | ||
| style="white-space: nowrap;" | --FileMove.DisableFileRename | | style="white-space: nowrap;" | --FileMove.DisableFileRename | ||
| style="white-space: nowrap;" | False | | style="white-space: nowrap;" | False | ||
Line 179: | Line 247: | ||
| style="white-space: nowrap;" | --Replacements | | style="white-space: nowrap;" | --Replacements | ||
| | | | ||
| | | Replace substrings in the returned filepath | ||
| style="white-space: nowrap;" | --FileMove.Replacements=<Match1>=<Replacement1>[;<Match2>=<Replacement2>...] | | style="white-space: nowrap;" | --FileMove.Replacements=<Match1>=<Replacement1>[;<Match2>=<Replacement2>...] | ||
| | | | ||
Line 197: | Line 265: | ||
| | | | ||
| Print calculated hashes in hexadecimal format to console | | Print calculated hashes in hexadecimal format to console | ||
Consumers must be specified for the the PrintHashes command to output anything | |||
| style="white-space: nowrap;" | --PrintHashes | | style="white-space: nowrap;" | --PrintHashes | ||
| style="white-space: nowrap;" | False | | style="white-space: nowrap;" | False | ||
Line 216: | Line 285: | ||
| Reports will be saved to the specified directory | | Reports will be saved to the specified directory | ||
| style="white-space: nowrap;" | --ReportDirectory=<Directory> | | style="white-space: nowrap;" | --ReportDirectory=<Directory> | ||
| style="white-space: nowrap;" | <The directory AVD3 | | style="white-space: nowrap;" | <The directory AVD3 is invoked in> | ||
|- | |- | ||
| style="white-space: nowrap;" | --ReportFileName | | style="white-space: nowrap;" | --ReportFileName | ||
| | | | ||
| Reports will be saved/appended to the specified filename<br /> | | Reports will be saved/appended to the specified filename<br />Placeholders mentioned in --FileMove.Pattern can be used as well.<br />Additional placeholders: ReportName, ReportFileExtension | ||
| style="white-space: nowrap;" | --ReportFileName=<FileName> | | style="white-space: nowrap;" | --ReportFileName=<FileName> | ||
| style="white-space: nowrap;" | ${FileName}.${ReportName}.${ReportFileExtension} | | style="white-space: nowrap;" | ${FileName}.${ReportName}.${ReportFileExtension} | ||
|- | |||
| style="white-space: nowrap;" | --ReportContentPrefix | |||
| | |||
| Each report will be prefixed with the arguments content.<br />See ReportFileName for Placeholders. | |||
| style="white-space: nowrap;" | --ReportContentPrefix=${FullName} | |||
| style="white-space: nowrap;" | | |||
|- | |- | ||
| style="white-space: nowrap;" | --ExtensionDifferencePath | | style="white-space: nowrap;" | --ExtensionDifferencePath | ||
Line 233: | Line 308: | ||
| | | | ||
| Searches the filename for the calculated CRC32 hash. If not present or different a line with the caluclated hash and the full path of the file is appended to the specified path<br />The regex pattern should contain the placeholder ${CRC32} which is replaced by the calculated hash prior matching.<br />Consumer CRC32 will be force enabled! | | Searches the filename for the calculated CRC32 hash. If not present or different a line with the caluclated hash and the full path of the file is appended to the specified path<br />The regex pattern should contain the placeholder ${CRC32} which is replaced by the calculated hash prior matching.<br />Consumer CRC32 will be force enabled! | ||
| style="white-space: nowrap;" | --CRC32Error=<Filepath> | | style="white-space: nowrap;" | --CRC32Error=<Filepath>,<RegexPattern> | ||
| style="white-space: nowrap;" | (, (?i)${CRC32}) | | style="white-space: nowrap;" | (, (?i)${CRC32}) | ||
|} | |} | ||
Line 246: | Line 321: | ||
! Usage | ! Usage | ||
! Default Value | ! Default Value | ||
|- | |||
| style="white-space: nowrap;" | --Version | |||
| | |||
| Print the program version to console | |||
| style="white-space: nowrap;" | --Version | |||
| style="white-space: nowrap;" | False | |||
|- | |- | ||
| style="white-space: nowrap;" | --SaveErrors | | style="white-space: nowrap;" | --SaveErrors | ||
Line 263: | Line 344: | ||
| Various places may include personal data. Currently this only affects error files, which will then include the full filepath | | Various places may include personal data. Currently this only affects error files, which will then include the full filepath | ||
| style="white-space: nowrap;" | --IncludePersonalData | | style="white-space: nowrap;" | --IncludePersonalData | ||
| style="white-space: nowrap;" | | | style="white-space: nowrap;" | False | ||
|- | |||
| style="white-space: nowrap;" | --PrintDiscoveredFiles | |||
| | |||
| Print each discovered file path instead of printing the count | |||
| style="white-space: nowrap;" | --PrintDiscoveredFiles | |||
| style="white-space: nowrap;" | <The directory AVD3 is invoked in> | |||
|- | |- | ||
| style="white-space: nowrap;" | --ErrorDirectory | | style="white-space: nowrap;" | --ErrorDirectory | ||
Line 269: | Line 356: | ||
| If --SaveErrors is specified the error files will be placed in the specified path | | If --SaveErrors is specified the error files will be placed in the specified path | ||
| style="white-space: nowrap;" | --ErrorDirectory=<DirectoryPath> | | style="white-space: nowrap;" | --ErrorDirectory=<DirectoryPath> | ||
| style="white-space: nowrap;" | <The directory AVD3 | | style="white-space: nowrap;" | <The directory AVD3 is invoked in> | ||
|- | |- | ||
| style="white-space: nowrap;" | --NullStreamTest | | style="white-space: nowrap;" | --NullStreamTest | ||
Line 275: | Line 362: | ||
| Use Memory as the DataSource for HashSpeed testing. Overrides any FileDiscovery Settings! | | Use Memory as the DataSource for HashSpeed testing. Overrides any FileDiscovery Settings! | ||
| style="white-space: nowrap;" | --NullStreamTest=<StreamCount>:<StreamLength in MiB>:<ParallelStreamCount> | | style="white-space: nowrap;" | --NullStreamTest=<StreamCount>:<StreamLength in MiB>:<ParallelStreamCount> | ||
| style="white-space: nowrap;" | | | style="white-space: nowrap;" | | ||
|} | |} | ||
Line 372: | Line 459: | ||
| style="white-space: nowrap;" | --UploadErrors | | style="white-space: nowrap;" | --UploadErrors | ||
| | | | ||
| Enables the automatic upload of program errors | | Enables the automatic upload of program errors. Please be aware that if --IncludePersonalData is enabled, personal data is uploaded as well! | ||
| | | style="white-space: nowrap;" | --UploadErrors | ||
| style="white-space: nowrap;" | False | | style="white-space: nowrap;" | False | ||
|- | |- | ||
Line 381: | Line 468: | ||
| style="white-space: nowrap;" | --Ed2KLogPath=<path> | | style="white-space: nowrap;" | --Ed2KLogPath=<path> | ||
| | | | ||
|- | |||
| style="white-space: nowrap;" | --PrintEd2kLink | |||
| | |||
| Prints the ED2K-Link after a file has been processed into the console.<br />Consumer ED2K will be force enabled! | |||
| style="white-space: nowrap;" | --PrintEd2kLink | |||
| style="white-space: nowrap;" | False | |||
|} | |} | ||
==Auto-creqing== | |||
[[Image:Autocreq.gif|thumb|Simple diagram of the way auto-creqing with Avdump3 works]] | |||
Avdump3 provides metadata for the AniDB [[auto-creqing]] system. Some more or less important notes: | |||
* To be able to use this feature you’ll need an AniDB account '''and you have to define the ''UDP API Key'' in your [[profile]]'''. | |||
* All data sent to the server will be logged with IP and uid. | |||
* There is no direct connection between data sent to AniDB and creqs generated. The data received will just be stored for later processing. | |||
* There is no way to check the current status for a dump. Usually, it should take ''at least 24 hours'' from the moment you dump a file till the data actually changes. If any irregularities occur, or when there is too much data pending, it will take more time. | |||
* You may dump files currently not in the database. The data is still stored and will be used later if/after the file has been registered. | |||
* The creqs generated will report the user who sent the data first (for a specific file) as the creqer. | |||
* Files creqed by the new system will be locked, meaning some fields will not be possible to change. Notify a moderator if you are '''sure''' that some of the data locked for a specific file is wrong. | |||
===Why didn't this file get dumped?=== | |||
* The file in AniDB is registered with wrong size and/or ED2K hash. | |||
* The package never reached the server. | |||
* The decryption or decompression failed at server side (rare). | |||
* The dump is not valid XML (rare). | |||
===Why isn't this file verified?=== | |||
Even if the file is dumped it doesn't mean it will get auto-creqed (and verified). Here are the reasons: | |||
* The file is corrupt/invalid. | |||
* The provided data is considered "incoherent". (Indicates a bug in Avdump.) | |||
* The dump was marked unfit for file verification by an AniDB Moderator. | |||
== Development == | == Development == | ||
Source code | Source code for the Core project can be found at GitHub: https://github.com/DvdKhl/AVDump3 | ||
The source code for the AniDB Module is not available to the public. | The source code for the AniDB Module is not available to the public. | ||
== Planned Features == | |||
=== Graphical User Interface === | |||
=== File moving/renaming with AniDB Http Api support === | |||
==Changelog== | |||
<div style="height: 280px; overflow: auto; padding: 3px; border:1px solid #AAAAAA; font-size: small"> | |||
'''0.1.8293.0 : 2021.12.02''' | |||
* Updated dependency from .Net 5.0 to .Net 6.0 | |||
* Added support for Linux aarch64 | |||
'''0.1.8238.0 : 2021.10.31''' | |||
* Updated dependency from .Net Core 3.1 to .Net 5.0 | |||
* Updated to MediaInfoLib 2109 | |||
* Fixed various small bugs | |||
'''0.1.8213.0 : 2020.09.27''' | |||
* Fixed ED2k hash calculation for files with a file size that is a multiple of 9728000 bytes (Issue #48) | |||
** Included alternative ED2k link in ed2k log | |||
** Added <ed2k_alt/> node to output xml | |||
* Don't crash when a file to be processed cannot be opened (Issue #53) | |||
* Switched from FileShare.Read to FileShare.ReadWrite for writing to files | |||
* Added TiB/s | |||
* Set UTF8 as Console output when UTF8OUT is added as an arg (console wide!) | |||
* Added PARSEARGS which will make AVD3 parse the provided arguments itself (windows only), fixing the double quote escape problem | |||
* Don't crash when the console cursor cannot be manipulated | |||
* Output final progress before termination | |||
* Fixed various small bugs | |||
'''0.1.8188.0 : 2020.08.23''' | |||
* Fixed various small bugs | |||
'''0.1.8187.0 : 2020.08.22''' | |||
* Fixed various small bugs | |||
'''0.1.8185.0 : 2020.08.17''' | |||
* Updated MIL to version 20.08 | |||
* Fixed various small bugs | |||
'''0.1.8173.0 : 2020.08.15''' | |||
* Fixed ReportFileName throwing an exception when used | |||
* EffectiveCommandLineArguments now with sub elements | |||
'''0.1.8167.0 : 2020.08.14''' | |||
* Fixed crash while displaying the progress | |||
* Switched to portable pdb (linenumbers in linux) | |||
* Implemented "UserValueType" for setting properties (Issue #45) | |||
** Passwords are now marked with PasswordType for UserValueType | |||
** Errors now report effective arguments when IncludePersonalData is enabled | |||
*** Properties with type Password are hidden | |||
'''0.1.8163.0 : 2020.08.11''' | |||
* Skip reporting PathTooLongException exception when generating a report (Issue #42) | |||
* Sanity check for ogg bitstreams (Issue #41) | |||
* Own class to write lines to text files instead of using File.AppendAllText (Issue #40) | |||
* Fixed Bar display exception (Issue #39) | |||
* Fixed Exception when generating AniDBReport (Issue #A23) | |||
* Fixed System.NullReferenceException when ED2K consumer was not enabled (Issue #A22) | |||
'''0.1.8152.0 : 2020.08.10''' | |||
* Use comma instead of colon for --CRC32Error (Issue #37) | |||
* Added suggestion when Mirrored Buffer cannot be created (Issue #36) | |||
* Unified FileMove.Pattern and ReportFileName Placeholders (Issue #35) | |||
* When the terminal window is too small (<72) the output went haywire (Issue #34) | |||
* Added --PrintEd2kLink (Issue #33) | |||
* Added --Version (Issue #32) | |||
'''0.1.8134.0 : 2020.08.09''' | |||
* semi-public pre-release | |||
</div> | |||
[[Category:Avdump]] |
Latest revision as of 03:06, 21 August 2023
AVDump is a tool to extract meta information from media files while at the same time calculating multiple hashes. Based on that information reports can be generated in multiple forms. Of particular interest is the ability to send those reports back to AniDB and thereby quickly filling in missing metadata for new files.
Quickstart
Important | Avdump GUI is a GUI version of Avdump3, for command line skeptics. |
Since you’re here you probably just want to send AVMF Packages (or Dumps) to AniDB which will add metadata to previously (or soon to be) added files. If that doesn’t mean anything to you, you probably want to start with Content:Files and Auto-creqing and Tutorial:How to Add Files for Dummies.
Anyway lets keep this short:
- Login into AniDB with your account and visit https://anidb.net/user/setting, there go to the Account tab and set a password for “UDP API Key”
- Make sure .Net 6.0 or higher is installed
- Download the latest Version of AVD3 with the link in the section below and extract it somewhere.
- Start a terminal and navigate to AVDump3
- Use the following arguments as the bare minimum
- Windows:
AVDump3CL.exe --Auth=<YourUserName>:<YourUdpApiPassword> <APathToTheFiles>
- Linux:
dotnet AVDump3CL.dll --Auth=<YourUserName>:<YourUdpApiPassword> <APathToTheFiles>
- Windows:
- Optional: See sections below to improve usage experience
- For example, to avoid dumping the same files over and over again, add the following parameter:
--DoneLogPath=done.txt
- For example, to avoid dumping the same files over and over again, add the following parameter:
- Optional: Strongly consider adding --UploadErrors to the arguments. For more information see https://wiki.anidb.net/Avdump3#UploadErrors_Argument
Download
DOWNLOAD (ZIP) (Build 8293) MD5 = 87ed86a9babc172d638071ff1123fdc7
What’s new compared to AVD2?
AVD3 is a complete rewrite of AVD2 which uses .NET 6.0 instead of .NET Framework and treats Linux as a first class citizen. So it should run on Linux just as well as on Windows. To name a few other big differences:
- Can process multiple files in parallel
- Uses native code to speed up hashing significantly
- More efficient reading
- Can move/rename files based on scripts
- Redone command line arguments
- More hash algorithms
- Latest version of MediaInfoLib (MIL) is being used (currently 2109)
- Support for .vtt subtitle files
- Support for 32bit has been dropped
- Support for Linux aarch64 has been added
- Support for MacOS is not yet available (To add support https://github.com/DvdKhl/AVDump3/blob/master/AVDump3NativeLib/src/AVD3MirrorBuffer.c needs to be implemented, help would be appreciated)
Supported formats
Most major file formats are supported to some degree. While hashes are created for all file types, stream details remain shaky for others (notably 'swf'). Because of that for some filetypes (underlined) only the hashes get auto-creqed.
- Video files: asf/wmv, avi, flv, m2ts, mk3d, mkv, mov, mp4, mpg/mpeg, ogm, ogv, qt, rm/rmvb, swf, ts, webm
- Subtitle files: ass, idx, js, lrc, mks, pjs, rt, smi, srt, ssa, sub, sup, tmp, tts, txt, vtt, xss
- Audio files: aac, ac3, dts, dtshd, flac, m4a, mka, mp3, ogg, ra, thd, wav, wma
- Archive files: 7z, ace, rar, zip
- Linker files: mkv, smil
UploadErrors Argument
When enabled errors while processing or program crashes are uploaded to AniDB which makes it far easier to discover and fix bugs.
Error reports sent look like the following:
<AVD3CLException thrownOn="2020-08-11 18:31:34.2837"> <Information> <EntryAssemblyVersion>3.0.8163.0</EntryAssemblyVersion> <LibVersion>3.0.8163.0</LibVersion> <Session>395e5893-145d-4a19-8fc9-1ddc0f7feed6</Session> <Framework>3.1.7</Framework> <OSVersion>Microsoft Windows NT 6.2.9200.0</OSVersion> <IntPtr.Size>8</IntPtr.Size> <Is64BitOperatingSystem>true</Is64BitOperatingSystem> <Is64BitProcess>true</Is64BitProcess> <ProcessorCount>40</ProcessorCount> <UserInteractive>true</UserInteractive> <SystemPageSize>4096</SystemPageSize> <WorkingSet>239337472</WorkingSet> </Information> <Message>CreatingInfoProviders</Message> <FileName>Hidden(7E940D28B4A58A87C4C19B39EB6EA5E2B11CC7B13762DE44BB7338E752F6676722B227DBA942302F956AF1CE040C57D9295CF53AB33B5DE45FD4FAB6FB462E47)</FileName> <Cause> <InvalidOperationException> <Message>MediaInfoLib couldn't open the file</Message> <Stacktrace> <Frame>at AVDump3Lib.Information.InfoProvider.MediaInfoLibProvider..ctor(String filePath) in D:\Projects\C#\AVDump3\AVDump3Lib\Information\InfoProvider\MediaInfoLibProvider.cs:line 522</Frame> <Frame>at AVDump3Lib.Information.AVD3InformationModule.<>c.<.ctor>b__4_4(InfoProviderSetup setup) in D:\Projects\C#\AVDump3\AVDump3Lib\Information\AVD3InformationModule.cs:line 30</Frame> <Frame>at AVDump3Lib.Information.InfoProvider.InfoProviderFactory.Create(InfoProviderSetup setup) in D:\Projects\C#\AVDump3\AVDump3Lib\Information\InfoProvider\InfoProviderFactory.cs:line 40</Frame> <Frame>at AVDump3CL.AVD3CLModule.<>c__DisplayClass34_0.<CreateFileMetaInfo>b__0(IInfoProviderFactory x) in D:\Projects\C#\AVDump3\AVDump3CL\AVD3CLModule.cs:line 449</Frame> <Frame>at System.Linq.Enumerable.SelectIListIterator`2.ToArray()</Frame> <Frame>at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)</Frame> <Frame>at AVDump3CL.AVD3CLModule.CreateFileMetaInfo(String filePath, ImmutableArray`1 blockConsumers) in D:\Projects\C#\AVDump3\AVDump3CL\AVD3CLModule.cs:line 449</Frame> </Stacktrace> </InvalidOperationException> </Cause> <Stacktrace /> </AVD3CLException>
The value within Hidden() is a hash so it is not possible to get the original data back from that value. When --IncludePersonalData is added those values will be shown in clear text.
Attention: Please be aware that when --IncludePersonalData is active as well, additional data like the commandline arguments (except for passwords) and path information is sent as well!
Arguments
For more detailed information please run AVD3 with --Help
!
Special Arguments
- FROMFILE
If the first argument is called FROMFILE and the next argument is a file path, every line of that file is interpreted as a single argument. A line which starts with // is interpreted as a comment and ignored. See https://wiki.anidb.net/Talk:Avdump3 for an example.
- PRINTARGS
Prints the effective arguments used into the terminal.
FileDiscovery
Parameter | Shorthand | Description | Usage | Default Value |
---|---|---|---|---|
--Recursive | -R | Recursively descent into Subdirectories | --Recursive | False |
--ProcessedLogPath | --PLPath | Appends the full filepath to the specified path | --ProcessedLogPath=<FilePath1>[:<FilePath2>...] | |
--SkipLogPath | --SLPath | Filepaths contained in the specified file will not be processed | --SkipLogPath=<FilePath1>[:<FilePath2>...] | |
--DoneLogPath | --DLPath | Will set --SkipLogPath and --ProcessedLogPath to the specified filepath | --DoneLogPath=<Filepath> | |
--WithExtensions | --WExts | Only/Don't Process files with selected Extensions | --WithExtensions=[-]<Extension1>[,<Extension2>...] | |
--Concurrent | --Conc | Sets the maximal number of files which will be processed concurrently. First param (max) sets a global limit. (path,max) pairs sets limits per path. | --Concurrent=<max>[:<path1>,<max1>;<path2>,<max2>...] | 1 |
Processing
Parameter | Shorthand | Description | Usage | Default Value |
---|---|---|---|---|
--ProducerMinReadLength | How much data in MiB the reader has to read each time at minimum | --ProducerMinReadLength | 1 | |
--ProducerMaxReadLength | How much data in MiB the reader is allowed to read each time at most | --ProducerMaxReadLength | 8 | |
--PrintAvailableSIMDs | Print available CPU SIMDs | --PrintAvailableSIMDs | False | |
--PauseBeforeExit | --PBExit | Pause console before exiting | --PauseBeforeExit | False |
--BufferLength | --BLength | Circular buffer size for hashing | --BufferLength=<Size in MiB> | 64 |
--Consumers | --Cons | Select consumers to use. Use without arguments to list available consumers | --Consumers=<ConsumerName1>[,<ConsumerName2>...] |
FileMove
Parameter | Shorthand | Description | Usage | Default Value |
---|---|---|---|---|
--Test | Test FileMove Settings | --FileMove.Test | False | |
--LogPath | A line is written for each file that has been moved/renamed. (OldPath => NewPath) | --FileMove.LogPath=<FilePath> | ||
--Mode | Determines how the Pattern Argument is going to be interpreted: Inline: Script is directly entered as the argument File: A path pointing to the script file Placeholder: See example for --Pattern CSharpScript: Script in C# DotNetAssembly: Net Core assembly to be loaded |
--FileMove.Mode=<None|PlaceholderInline|PlaceholderFile|CSharpScriptInline|CSharpScriptFile|DotNetAssembly> | None | |
--Pattern | Available Placeholders ${Name}: FileSize, FullName, FileName, FileExtension, FileNameWithoutExtension, DirectoryName, SuggestedExtension, Hash-<Name>-<2|4|8|10|16|32|32Hex|32Z|36|62|64>-<OC|UC|LC> |
--FileMove.Pattern=${DirectoryName}\${FileNameWithoutExtension}${SuggestedExtension} | ${DirectoryName}\${FileNameWithoutExtension}${FileExtension} | |
--DisableFileMove | Don't move the file even if the Pattern says so | --FileMove.DisableFileMove | False | |
--DisableFileRename | Don't rename the file even if the Pattern says so | --FileMove.DisableFileRename | False | |
--Replacements | Replace substrings in the returned filepath | --FileMove.Replacements=<Match1>=<Replacement1>[;<Match2>=<Replacement2>...] |
Reporting
Parameter | Shorthand | Description | Usage | Default Value |
---|---|---|---|---|
--PrintHashes | Print calculated hashes in hexadecimal format to console
Consumers must be specified for the the PrintHashes command to output anything |
--PrintHashes | False | |
--PrintReports | Print generated reports to console | --PrintReports | False | |
--Reports | Select reports to use. Use without arguments to list available reports | --Reports | ||
--ReportDirectory | --RDir | Reports will be saved to the specified directory | --ReportDirectory=<Directory> | <The directory AVD3 is invoked in> |
--ReportFileName | Reports will be saved/appended to the specified filename Placeholders mentioned in --FileMove.Pattern can be used as well. Additional placeholders: ReportName, ReportFileExtension |
--ReportFileName=<FileName> | ${FileName}.${ReportName}.${ReportFileExtension} | |
--ReportContentPrefix | Each report will be prefixed with the arguments content. See ReportFileName for Placeholders. |
--ReportContentPrefix=${FullName} | ||
--ExtensionDifferencePath | --EDPath | Logs the filepath if the detected extension does not match the actual extension | --EDPath=extdiff.txt | |
--CRC32Error | Searches the filename for the calculated CRC32 hash. If not present or different a line with the caluclated hash and the full path of the file is appended to the specified path The regex pattern should contain the placeholder ${CRC32} which is replaced by the calculated hash prior matching. Consumer CRC32 will be force enabled! |
--CRC32Error=<Filepath>,<RegexPattern> | (, (?i)${CRC32}) |
Diagnostics
Parameter | Shorthand | Description | Usage | Default Value |
---|---|---|---|---|
--Version | Print the program version to console | --Version | False | |
--SaveErrors | Errors occuring during program execution will be saved to disk | --SaveErrors | False | |
--SkipEnvironmentElement | Skip the environment element in error files | --SkipEnvironmentElement | False | |
--IncludePersonalData | Various places may include personal data. Currently this only affects error files, which will then include the full filepath | --IncludePersonalData | False | |
--PrintDiscoveredFiles | Print each discovered file path instead of printing the count | --PrintDiscoveredFiles | <The directory AVD3 is invoked in> | |
--ErrorDirectory | If --SaveErrors is specified the error files will be placed in the specified path | --ErrorDirectory=<DirectoryPath> | <The directory AVD3 is invoked in> | |
--NullStreamTest | Use Memory as the DataSource for HashSpeed testing. Overrides any FileDiscovery Settings! | --NullStreamTest=<StreamCount>:<StreamLength in MiB>:<ParallelStreamCount> |
Display
Parameter | Shorthand | Description | Usage | Default Value |
---|---|---|---|---|
--HideBuffers | Hides buffer bars | --HideBuffers | False | |
--HideFileProgress | Hides file progress | --HideFileProgress | False | |
--HideTotalProgress | Hides total progress | --HideTotalProgress | False | |
--ShowDisplayJitter | Displays the time taken to calculate progression stats and drawing to console | --ShowDisplayJitter | False | |
--ForwardConsoleCursorOnly | The cursor position of the console will not be explicitly set. This option will disable most progress output | --ForwardConsoleCursorOnly | False |
AniDBAvmf
Parameter | Shorthand | Description | Usage | Default Value |
---|---|---|---|---|
--LocalPort | --LPort | Local UDP port used for ACReqing | --LPort=<localport> | |
--ACreqErrorPath | A line is added to the specified file for every error which occurred during an Avmf Package transmission | --ACreqErrorPath=<FilePath> | ||
--Authentication | --Auth | Enables ACReqing when valid credentials are provided Visit https://anidb.net/user/setting (Account Tab) to set the api key |
--Auth=<username>:<api_key> | |
--HostEndPoint | --Host | Change endpoint of AniDB UDP API server AddressFamily: Can be 4=IPv4, 6=IPv6 or U=Unspecified |
--Host=<hostname>:<hostport>[:<AddressFamily>] | api.anidb.info:9002:U |
--Timeout | --TOut | Sets the retry count and the timeout before resending the dump | --TOut=<seconds>:<retries> | 20:3 |
AniDBMisc
Parameter | Shorthand | Description | Usage | Default Value |
---|---|---|---|---|
--UploadErrors | Enables the automatic upload of program errors. Please be aware that if --IncludePersonalData is enabled, personal data is uploaded as well! | --UploadErrors | False | |
--Ed2kLogPath | Appends the ED2K-Link after a file has been processed into specified file separated by a line feed character (\n) Consumer ED2K will be force enabled! |
--Ed2KLogPath=<path> | ||
--PrintEd2kLink | Prints the ED2K-Link after a file has been processed into the console. Consumer ED2K will be force enabled! |
--PrintEd2kLink | False |
Auto-creqing
Avdump3 provides metadata for the AniDB auto-creqing system. Some more or less important notes:
- To be able to use this feature you’ll need an AniDB account and you have to define the UDP API Key in your profile.
- All data sent to the server will be logged with IP and uid.
- There is no direct connection between data sent to AniDB and creqs generated. The data received will just be stored for later processing.
- There is no way to check the current status for a dump. Usually, it should take at least 24 hours from the moment you dump a file till the data actually changes. If any irregularities occur, or when there is too much data pending, it will take more time.
- You may dump files currently not in the database. The data is still stored and will be used later if/after the file has been registered.
- The creqs generated will report the user who sent the data first (for a specific file) as the creqer.
- Files creqed by the new system will be locked, meaning some fields will not be possible to change. Notify a moderator if you are sure that some of the data locked for a specific file is wrong.
Why didn't this file get dumped?
- The file in AniDB is registered with wrong size and/or ED2K hash.
- The package never reached the server.
- The decryption or decompression failed at server side (rare).
- The dump is not valid XML (rare).
Why isn't this file verified?
Even if the file is dumped it doesn't mean it will get auto-creqed (and verified). Here are the reasons:
- The file is corrupt/invalid.
- The provided data is considered "incoherent". (Indicates a bug in Avdump.)
- The dump was marked unfit for file verification by an AniDB Moderator.
Development
Source code for the Core project can be found at GitHub: https://github.com/DvdKhl/AVDump3
The source code for the AniDB Module is not available to the public.
Planned Features
Graphical User Interface
File moving/renaming with AniDB Http Api support
Changelog
0.1.8293.0 : 2021.12.02
- Updated dependency from .Net 5.0 to .Net 6.0
- Added support for Linux aarch64
0.1.8238.0 : 2021.10.31
- Updated dependency from .Net Core 3.1 to .Net 5.0
- Updated to MediaInfoLib 2109
- Fixed various small bugs
0.1.8213.0 : 2020.09.27
- Fixed ED2k hash calculation for files with a file size that is a multiple of 9728000 bytes (Issue #48)
- Included alternative ED2k link in ed2k log
- Added <ed2k_alt/> node to output xml
- Don't crash when a file to be processed cannot be opened (Issue #53)
- Switched from FileShare.Read to FileShare.ReadWrite for writing to files
- Added TiB/s
- Set UTF8 as Console output when UTF8OUT is added as an arg (console wide!)
- Added PARSEARGS which will make AVD3 parse the provided arguments itself (windows only), fixing the double quote escape problem
- Don't crash when the console cursor cannot be manipulated
- Output final progress before termination
- Fixed various small bugs
0.1.8188.0 : 2020.08.23
- Fixed various small bugs
0.1.8187.0 : 2020.08.22
- Fixed various small bugs
0.1.8185.0 : 2020.08.17
- Updated MIL to version 20.08
- Fixed various small bugs
0.1.8173.0 : 2020.08.15
- Fixed ReportFileName throwing an exception when used
- EffectiveCommandLineArguments now with sub elements
0.1.8167.0 : 2020.08.14
- Fixed crash while displaying the progress
- Switched to portable pdb (linenumbers in linux)
- Implemented "UserValueType" for setting properties (Issue #45)
- Passwords are now marked with PasswordType for UserValueType
- Errors now report effective arguments when IncludePersonalData is enabled
- Properties with type Password are hidden
0.1.8163.0 : 2020.08.11
- Skip reporting PathTooLongException exception when generating a report (Issue #42)
- Sanity check for ogg bitstreams (Issue #41)
- Own class to write lines to text files instead of using File.AppendAllText (Issue #40)
- Fixed Bar display exception (Issue #39)
- Fixed Exception when generating AniDBReport (Issue #A23)
- Fixed System.NullReferenceException when ED2K consumer was not enabled (Issue #A22)
0.1.8152.0 : 2020.08.10
- Use comma instead of colon for --CRC32Error (Issue #37)
- Added suggestion when Mirrored Buffer cannot be created (Issue #36)
- Unified FileMove.Pattern and ReportFileName Placeholders (Issue #35)
- When the terminal window is too small (<72) the output went haywire (Issue #34)
- Added --PrintEd2kLink (Issue #33)
- Added --Version (Issue #32)
0.1.8134.0 : 2020.08.09
- semi-public pre-release