Avdump3: Difference between revisions

6,190 bytes added ,  2 October 2020
No edit summary
(39 intermediate revisions by 3 users not shown)
Line 14: Line 14:
#: '''Linux:''' <code>dotnet AVDump3CL.dll --Auth=&lt;YourUserName&gt;:&lt;YourUdpApiPassword&gt; &lt;APathToTheFiles&gt;</code>
#: '''Linux:''' <code>dotnet AVDump3CL.dll --Auth=&lt;YourUserName&gt;:&lt;YourUdpApiPassword&gt; &lt;APathToTheFiles&gt;</code>
# Optional: See sections below to improve usage experience
# Optional: See sections below to improve usage experience
## For example, to avoid dumping the same files over and over again, add the following parameter: <code>--DoneLogPath=done.txt</code>
# Optional: Strongly consider adding --UploadErrors to the arguments. For more information see https://wiki.anidb.net/Avdump3#UploadErrors_Argument


== Download ==
== Download ==


(http://static.anidb.net/client/avdump3/avdump3_8134.zip DOWNLOAD (ZIP)) (Build 8134) MD5 = 88264b3b4df756a3a2d4f79ad60dd622
[http://static.anidb.net/client/avdump3/avdump3_8188_beta.zip BETA DOWNLOAD (ZIP)] (Build 8188) MD5 = 3720b40762d322abed66f478adc1cefe


== What’s new compared to AVD2? ==
== What’s new compared to AVD2? ==
Line 29: Line 31:
* Redone commandline arguments  
* Redone commandline arguments  
* More hash algorithms
* More hash algorithms
* Latest version of MediaInfoLib (MIL) is being used (currently 20.03)
* Latest version of MediaInfoLib (MIL) is being used (currently 20.08)
* Support for .vtt subtitle files
* Support for .vtt subtitle files
* Support for 32bit has been dropped
* Support for 32bit has been dropped
Line 42: Line 44:
* '''Archive files:''' <u>7z</u>, <u>ace</u>, <u>rar</u>, <u>zip</u>
* '''Archive files:''' <u>7z</u>, <u>ace</u>, <u>rar</u>, <u>zip</u>
* '''Linker files:''' mkv, smil
* '''Linker files:''' mkv, smil
== UploadErrors Argument ==
When enabled errors while processing or program crashes are uploaded to AniDB which makes it far easier to discover and fix bugs.
Error reports sent look like the following:
<AVD3CLException thrownOn="2020-08-11 18:31:34.2837">
    <Information>
        <EntryAssemblyVersion>3.0.8163.0</EntryAssemblyVersion>
        <LibVersion>3.0.8163.0</LibVersion>
        <Session>395e5893-145d-4a19-8fc9-1ddc0f7feed6</Session>
        <Framework>3.1.7</Framework>
        <OSVersion>Microsoft Windows NT 6.2.9200.0</OSVersion>
        <IntPtr.Size>8</IntPtr.Size>
        <Is64BitOperatingSystem>true</Is64BitOperatingSystem>
        <Is64BitProcess>true</Is64BitProcess>
        <ProcessorCount>40</ProcessorCount>
        <UserInteractive>true</UserInteractive>
        <SystemPageSize>4096</SystemPageSize>
        <WorkingSet>239337472</WorkingSet>
    </Information>
    <Message>CreatingInfoProviders</Message>
    <Data>
        <FileName>Hidden(7E940D28B4A58A87C4C19B39EB6EA5E2B11CC7B13762DE44BB7338E752F6676722B227DBA942302F956AF1CE040C57D9295CF53AB33B5DE45FD4FAB6FB462E47)</FileName>
    </Data>
    <Cause>
        <InvalidOperationException>
            <Message>MediaInfoLib couldn't open the file</Message>
            <Stacktrace>
                <Frame>at AVDump3Lib.Information.InfoProvider.MediaInfoLibProvider..ctor(String filePath) in D:\Projects\C#\AVDump3\AVDump3Lib\Information\InfoProvider\MediaInfoLibProvider.cs:line 522</Frame>
                <Frame>at AVDump3Lib.Information.AVD3InformationModule.&lt;&gt;c.&lt;.ctor&gt;b__4_4(InfoProviderSetup setup) in D:\Projects\C#\AVDump3\AVDump3Lib\Information\AVD3InformationModule.cs:line 30</Frame>
                <Frame>at AVDump3Lib.Information.InfoProvider.InfoProviderFactory.Create(InfoProviderSetup setup) in D:\Projects\C#\AVDump3\AVDump3Lib\Information\InfoProvider\InfoProviderFactory.cs:line 40</Frame>
                <Frame>at AVDump3CL.AVD3CLModule.&lt;&gt;c__DisplayClass34_0.&lt;CreateFileMetaInfo&gt;b__0(IInfoProviderFactory x) in D:\Projects\C#\AVDump3\AVDump3CL\AVD3CLModule.cs:line 449</Frame>
                <Frame>at System.Linq.Enumerable.SelectIListIterator`2.ToArray()</Frame>
                <Frame>at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)</Frame>
                <Frame>at AVDump3CL.AVD3CLModule.CreateFileMetaInfo(String filePath, ImmutableArray`1 blockConsumers) in D:\Projects\C#\AVDump3\AVDump3CL\AVD3CLModule.cs:line 449</Frame>
            </Stacktrace>
            <Data />
        </InvalidOperationException>
    </Cause>
    <Stacktrace />
</AVD3CLException>
The value within Hidden() is a hash so it is not possible to get the original data back from that value. When --IncludePersonalData is added those values will be shown in clear text.
'''Attention:''' Please be aware that when --IncludePersonalData is active as well, additional data like the commandline arguments (except for passwords) and path information is sent as well!


== Arguments ==
== Arguments ==
Line 181: Line 228:
| style="white-space: nowrap;" | --Pattern
| style="white-space: nowrap;" | --Pattern
|  
|  
| Available Placeholders ${Name}:<br &/>FullName, FileName, FileExtension, FileNameWithoutExtension, DirectoryName, SuggestedExtension,<br /><nowiki>Hash-<Name>-<2|4|8|10|16|32|32Hex|32Z|36|62|64>-<OC|UC|LC></nowiki>
| Available Placeholders ${Name}:<br &/>FileSize, FullName, FileName, FileExtension, FileNameWithoutExtension, DirectoryName, SuggestedExtension,<br /><nowiki>Hash-<Name>-<2|4|8|10|16|32|32Hex|32Z|36|62|64>-<OC|UC|LC></nowiki>
| style="white-space: nowrap;" | --FileMove.Pattern=${DirectoryName}\${FileNameWithoutExtension}${SuggestedExtension}
| style="white-space: nowrap;" | --FileMove.Pattern=${DirectoryName}\${FileNameWithoutExtension}${SuggestedExtension}
| style="white-space: nowrap;" | ${DirectoryName}\${FileNameWithoutExtension}${FileExtension}
| style="white-space: nowrap;" | ${DirectoryName}\${FileNameWithoutExtension}${FileExtension}
Line 236: Line 283:
| Reports will be saved to the specified directory
| Reports will be saved to the specified directory
| style="white-space: nowrap;" | --ReportDirectory=<Directory>
| style="white-space: nowrap;" | --ReportDirectory=<Directory>
| style="white-space: nowrap;" | <The directory AVD3 resides in>
| style="white-space: nowrap;" | <The directory AVD3 is invoked in>
|-
|-
| style="white-space: nowrap;" | --ReportFileName
| style="white-space: nowrap;" | --ReportFileName
|  
|  
| Reports will be saved/appended to the specified filename<br />The following placeholders ${Name} can be used: FileName, FileNameWithoutExtension, FileExtension, ReportName, ReportFileExtension
| Reports will be saved/appended to the specified filename<br />Placeholders mentioned in --FileMove.Pattern can be used as well.<br />Additional placeholders: ReportName, ReportFileExtension
| style="white-space: nowrap;" | --ReportFileName=<FileName>
| style="white-space: nowrap;" | --ReportFileName=<FileName>
| style="white-space: nowrap;" | ${FileName}.${ReportName}.${ReportFileExtension}
| style="white-space: nowrap;" | ${FileName}.${ReportName}.${ReportFileExtension}
Line 253: Line 300:
|  
|  
| Searches the filename for the calculated CRC32 hash. If not present or different a line with the caluclated hash and the full path of the file is appended to the specified path<br />The regex pattern should contain the placeholder ${CRC32} which is replaced by the calculated hash prior matching.<br />Consumer CRC32 will be force enabled!
| Searches the filename for the calculated CRC32 hash. If not present or different a line with the caluclated hash and the full path of the file is appended to the specified path<br />The regex pattern should contain the placeholder ${CRC32} which is replaced by the calculated hash prior matching.<br />Consumer CRC32 will be force enabled!
| style="white-space: nowrap;" | --CRC32Error=<Filepath>:<RegexPattern>
| style="white-space: nowrap;" | --CRC32Error=<Filepath>,<RegexPattern>
| style="white-space: nowrap;" | (, (?i)${CRC32})
| style="white-space: nowrap;" | (, (?i)${CRC32})
|}
|}
Line 266: Line 313:
! Usage
! Usage
! Default Value
! Default Value
|-
| style="white-space: nowrap;" | --Version
|
| Print the program version to console
| style="white-space: nowrap;" | --Version
| style="white-space: nowrap;" | False
|-
|-
| style="white-space: nowrap;" | --SaveErrors
| style="white-space: nowrap;" | --SaveErrors
Line 289: Line 342:
| If --SaveErrors is specified the error files will be placed in the specified path
| If --SaveErrors is specified the error files will be placed in the specified path
| style="white-space: nowrap;" | --ErrorDirectory=<DirectoryPath>
| style="white-space: nowrap;" | --ErrorDirectory=<DirectoryPath>
| style="white-space: nowrap;" | <The directory AVD3 resides in>
| style="white-space: nowrap;" | <The directory AVD3 is invoked in>
|-
|-
| style="white-space: nowrap;" | --NullStreamTest
| style="white-space: nowrap;" | --NullStreamTest
Line 295: Line 348:
| Use Memory as the DataSource for HashSpeed testing. Overrides any FileDiscovery Settings!
| Use Memory as the DataSource for HashSpeed testing. Overrides any FileDiscovery Settings!
| style="white-space: nowrap;" | --NullStreamTest=<StreamCount>:<StreamLength in MiB>:<ParallelStreamCount>
| style="white-space: nowrap;" | --NullStreamTest=<StreamCount>:<StreamLength in MiB>:<ParallelStreamCount>
| style="white-space: nowrap;" | 0:0:0
| style="white-space: nowrap;" |  
|}
|}


Line 401: Line 454:
| style="white-space: nowrap;" | --Ed2KLogPath=<path>
| style="white-space: nowrap;" | --Ed2KLogPath=<path>
|  
|  
|-
| style="white-space: nowrap;" | --PrintEd2kLink
|
| Prints the ED2K-Link after a file has been processed into the console.<br />Consumer ED2K will be force enabled!
| style="white-space: nowrap;" | --PrintEd2kLink
| style="white-space: nowrap;" | False
|}
|}


==Auto-creqing==
==Auto-creqing==
[[Image:Autocreq.gif|thumb|Simple diagram of the way auto-creqing with Avdump2 works]]
[[Image:Autocreq.gif|thumb|Simple diagram of the way auto-creqing with Avdump3 works]]


Avdump2 provides metadata for the AniDB [[auto-creqing]] system. Some more or less important notes:
Avdump3 provides metadata for the AniDB [[auto-creqing]] system. Some more or less important notes:
* To be able to use this feature you’ll need an AniDB account '''and you have to define the ''UDP API Key'' in your [[profile]]'''.  
* To be able to use this feature you’ll need an AniDB account '''and you have to define the ''UDP API Key'' in your [[profile]]'''.  
* All data sent to the server will be logged with IP and uid.
* All data sent to the server will be logged with IP and uid.
Line 426: Line 485:
* The provided data is considered "incoherent". (Indicates a bug in Avdump.)
* The provided data is considered "incoherent". (Indicates a bug in Avdump.)
* The dump was marked unfit for file verification by an AniDB Moderator.
* The dump was marked unfit for file verification by an AniDB Moderator.


== Development ==
== Development ==
Line 437: Line 494:
=== Graphical User Interface ===
=== Graphical User Interface ===
=== File moving/renaming with AniDB Http Api support ===
=== File moving/renaming with AniDB Http Api support ===
==Changelog==
<div style="height: 280px; overflow: auto; padding: 3px; border:1px solid #AAAAAA; font-size: small">
'''0.1.8213.0 : 2020.09.27'''
* Fixed ED2k hash calculation for files with a file size that is a multiple of 9728000 bytes (Issue #48)
** Included alternative ED2k link in ed2k log
** Added <ed2k_alt/> node to output xml
* Don't crash when a file to be processed cannot be opened (Issue #53)
* Switched from FileShare.Read to FileShare.ReadWrite for writing to files
* Added TiB/s
* Set UTF8 as Console output when UTF8OUT is added as an arg (console wide!)
* Added PARSEARGS which will make AVD3 parse the provided arguments itself (windows only), fixing the double quote escape problem
* Don't crash when the console cursor cannot be manipulated
* Output final progress before termination
* Fixed various small bugs
'''0.1.8188.0 : 2020.08.23'''
* Fixed various small bugs
'''0.1.8187.0 : 2020.08.22'''
* Fixed various small bugs
'''0.1.8185.0 : 2020.08.17'''
* Updated MIL to version 20.08
* Fixed various small bugs
'''0.1.8173.0 : 2020.08.15'''
* Fixed ReportFileName throwing an exception when used
* EffectiveCommandLineArguments now with sub elements
'''0.1.8167.0 : 2020.08.14'''
* Fixed crash while displaying the progress
* Switched to portable pdb (linenumbers in linux)
* Implemented "UserValueType" for setting properties (Issue #45)
** Passwords are now marked with PasswordType for UserValueType
** Errors now report effective arguments when IncludePersonalData is enabled
*** Properties with type Password are hidden
'''0.1.8163.0 : 2020.08.11'''
* Skip reporting PathTooLongException exception when generating a report (Issue #42)
* Sanity check for ogg bitstreams (Issue #41)
* Own class to write lines to text files instead of using File.AppendAllText (Issue #40)
* Fixed Bar display exception (Issue #39)
* Fixed Exception when generating AniDBReport (Issue #A23)
* Fixed System.NullReferenceException when ED2K consumer was not enabled (Issue #A22)
'''0.1.8152.0 : 2020.08.10'''
* Use comma instead of colon for --CRC32Error (Issue #37)
* Added suggestion when Mirrored Buffer cannot be created (Issue #36)
* Unified FileMove.Pattern and ReportFileName Placeholders (Issue #35)
* When the terminal window is too small (<72) the output went haywire (Issue #34)
* Added --PrintEd2kLink (Issue #33)
* Added --Version (Issue #32)
'''0.1.8134.0 : 2020.08.09'''
* semi-public pre-release
</div>
MediaWiki spam blocked by CleanTalk.
MediaWiki spam blocked by CleanTalk.