Avdump2 on Linux: Difference between revisions
m (Removed forgotten debug line from the example script) |
({{Avdump-current-version}}) |
||
(7 intermediate revisions by 5 users not shown) | |||
Line 1: | Line 1: | ||
{{TOCright}} | |||
{{eyecatch|WARNING|This article was written for [[Avdump2]]. The current latest version is {{Avdump-current-version}}, and the steps in this article have not yet been explicitly verified against {{Avdump-current-version}}.}} | |||
[[Avdump2]] was developed on Windows, for Windows. Thanks to Mono it’s possible to run Avdump2 on Linux and all efforts are made to support all features on Linux as well, but handling every peculiarity of every Mono version on every Linux installation is beyond the scope of Avdump2 at this time. Therefore, some things just don’t work on Linux, or at least not right away, without some workarounds. | [[Avdump2]] was developed on Windows, for Windows. Thanks to Mono it’s possible to run Avdump2 on Linux and all efforts are made to support all features on Linux as well, but handling every peculiarity of every Mono version on every Linux installation is beyond the scope of Avdump2 at this time. Therefore, some things just don’t work on Linux, or at least not right away, without some workarounds. | ||
Line 7: | Line 9: | ||
=== Mono === | === Mono === | ||
Avdump2 has successfully been tested with Mono 2.4 on Ubuntu | Avdump2 has successfully been tested with Mono 2.4 on Ubuntu, with Mono 2.8 on Gentoo, and Mono 5.20 on openSUSE Tumbleweed. | ||
'''<tt>Avdump2CL.exe</tt> looks like a Windows .exe, how do I start it?''' | '''<tt>Avdump2CL.exe</tt> looks like a Windows .exe, how do I start it?''' | ||
Line 16: | Line 18: | ||
On 32 bit systems Avdump2 sometimes dies with that error when processing many files at once due to heap memory fragmentation. If you use a donelog you can just restart Avdump2 and it’ll continue from where it crashed. | On 32 bit systems Avdump2 sometimes dies with that error when processing many files at once due to heap memory fragmentation. If you use a donelog you can just restart Avdump2 and it’ll continue from where it crashed. | ||
'''Mono complains about not being able to load type 'AVDump2CL.CL'''' | |||
The actual error message probably looks similar to this: | |||
<tt>Unhandled Exception: System.TypeLoadException: Could not load type 'AVDump2CL.CL' from assembly 'AVDump2CL, Version=0.1.6438.0, Culture=neutral, PublicKeyToken=null'.</tt> | |||
If you get this error then you probably don’t have all necessary mono packages installed. On a Debian-based system make sure that you have the packages <tt>libmono-corlib2.0-cil</tt> and <tt>libmono-system2.0-cil</tt> installed. | |||
=== libMediaInfo === | === libMediaInfo === | ||
Line 43: | Line 53: | ||
If you have a file with bogus characters in its name, even changing the locale won’t help. You have to rename the file to something sensible. | If you have a file with bogus characters in its name, even changing the locale won’t help. You have to rename the file to something sensible. | ||
=== dotnet === | |||
Avdump2 also works on Linux via '''dotnet'''; it is about 30% faster than running it via '''mono''', but appears to work with ASCII filenames only. | |||
Create a <tt>AVDump2CL.runtimeconfig.json</tt> in the folder that contains '''AVDump2CL.exe''' with the following content: | |||
<pre>{ | |||
"runtimeOptions": { | |||
"configProperties": { | |||
"System.GC.Server": true, | |||
"System.GC.Concurrent": true, | |||
"System.Threading.ThreadPool.MinThreads": 6, | |||
"System.Threading.ThreadPool.MaxThreads": 24 | |||
}, | |||
"tfm": "netcoreapp3.1", | |||
"framework": { | |||
"name": "Microsoft.NETCore.App", | |||
"version": "3.1.1" | |||
}, | |||
"applyPatches": true, | |||
"rollForwardOnNoCandidateFx": 1 | |||
} | |||
}</pre> | |||
Move the Linux lib to the DLL filename so '''dotnet''' finds it | |||
<pre>mv MediaInfo_x64.dll MediaInfo_x64.dll.windows | |||
cp -a libMediaInfo_x64.so MediaInfo_x64.dll | |||
</pre> | |||
== Workarounds == | == Workarounds == | ||
Line 63: | Line 99: | ||
export LD_LIBRARY_PATH=${AVDUMP2_HOME} | export LD_LIBRARY_PATH=${AVDUMP2_HOME} | ||
# fix bug with ncurses 6.1+ | |||
# mono/issues/6752 | |||
export TERM=xterm | |||
# If libMediaInfo still can't be found, uncomment this. | # If libMediaInfo still can't be found, uncomment this. | ||
Line 83: | Line 123: | ||
setterm -cursor on | setterm -cursor on | ||
fi | fi | ||
</pre> | |||
== Making Your Life Easier == | |||
When you made sure everything works fine, if you just want to easily hash files and generate ED2K links for adding new files, you could use this roughly-made Python 2 script. (best paired with Nautilus or Nemo actions) | |||
Please note that this script relies on Zenity extension to display the dialog containing the generated ED2K links. If you get an error running this script, please make sure you have it installed. | |||
<pre> | |||
#!/usr/bin/python | |||
import sys, os, subprocess | |||
### EDIT THE FOLLOWING ACCORDING TO YOUR INFO | |||
AVDUMP2CL_PATH = "/path/to/your/AVDump2CL.exe" | |||
USERNAME = " " | |||
API_KEY = " " | |||
### | |||
command = "mono %s --Auth=%s:%s --PrintEd2kLink -q" % (AVDUMP2CL_PATH, USERNAME, API_KEY) | |||
for arg in sys.argv[1:]: | |||
command += " \"%s\"" % arg | |||
output = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) | |||
output = output.stdout.readlines() | |||
ed2k = "" | |||
for line in output: | |||
if line[0:4] == "ed2k": | |||
ed2k += line | |||
os.system("zenity --info --text '%s'" % ed2k.rstrip()) | |||
</pre> | </pre> | ||
Line 97: | Line 168: | ||
Avdump2 exits with a return code of 0 on some errors, so don’t rely on the return code. It exits with a non-zero code on most serious errors, though. | Avdump2 exits with a return code of 0 on some errors, so don’t rely on the return code. It exits with a non-zero code on most serious errors, though. | ||
[[Category:Features]] | |||
[[Category:AniDB Clients]] | |||
[[Category:Avdump]] | |||
[[Category:Development]] |
Latest revision as of 01:25, 19 December 2021
WARNING | This article was written for Avdump2. The current latest version is Avdump3, and the steps in this article have not yet been explicitly verified against Avdump3. |
Avdump2 was developed on Windows, for Windows. Thanks to Mono it’s possible to run Avdump2 on Linux and all efforts are made to support all features on Linux as well, but handling every peculiarity of every Mono version on every Linux installation is beyond the scope of Avdump2 at this time. Therefore, some things just don’t work on Linux, or at least not right away, without some workarounds.
This page is a collection of various workarounds and suggestions that should help you get started with Avdump2 on Linux. If you need more assistance, try to catch CatWEax on IRC.
The Basics
Can't get Avdump2 to work at all? Here are some basic suggestions to set up Adump2.
Mono
Avdump2 has successfully been tested with Mono 2.4 on Ubuntu, with Mono 2.8 on Gentoo, and Mono 5.20 on openSUSE Tumbleweed.
Avdump2CL.exe looks like a Windows .exe, how do I start it?
Try making the file executable. If that doesn’t work, call it using the mono Interpreter: mono Avdump2CL.exe
OutOfMemory exceptions
On 32 bit systems Avdump2 sometimes dies with that error when processing many files at once due to heap memory fragmentation. If you use a donelog you can just restart Avdump2 and it’ll continue from where it crashed.
Mono complains about not being able to load type 'AVDump2CL.CL'
The actual error message probably looks similar to this:
Unhandled Exception: System.TypeLoadException: Could not load type 'AVDump2CL.CL' from assembly 'AVDump2CL, Version=0.1.6438.0, Culture=neutral, PublicKeyToken=null'.
If you get this error then you probably don’t have all necessary mono packages installed. On a Debian-based system make sure that you have the packages libmono-corlib2.0-cil and libmono-system2.0-cil installed.
libMediaInfo
Avdump2 only works with the libMediaInfo it comes with. If it can’t find that you will usually get MediaInfoLib is missing. It might start with a different version from elsewhere on your system, but then you’ll run into odd errors when you try to dump a file.
There are many possible causes if the library can’t be found.
Library search path missing
Most likely Mono can’t find the binary library even if it’s in the same directory as Avdump2CL.exe. Then you’ll have to set the environment variable LD_LIBRARY_PATH to point to the directory into which you extracted Avdump2. Write yourself a wrapper script to do that or have a look at the example wrapper scripts below. Sometimes you’ll also have to set MONO_PATH to point to the same directory.
Incompatible GLIBC
If your Linux installation is very old or you’re using a distribution that ships with an outdated GLIBC (CentOS is a prime offender here), Mono can’t link against libMediaInfo and Avdump2 will still complain about missing the library. To check if that’s the case, use ldd libMediaInfo*.so. If you get a message like version 'GLIBCXX_3.4.9' not found, you’re out of luck. You can either try to update GLIBC, which usually involves pain, or upgrade or change your distribution to one with a newer GLIBC.
Hashing works but I can’t dump anything
If you modified any of the required files Avdump2 came with, you will either get a version error or several timeouts and no meaningful error message. So no, you can’t edit the config file either.
Failed to create MediaInfoProvider
Failed to create MediaInfoProvider
Invalid value returned by mil. This error may indicate that your LANG or LC_ALL environment variables are not set to use UTF-8 encoding.
This error appears if you try to process a file with special characters in its name and your locale settings don’t match the file system encoding. Nowadays almost all file names are encoded with UTF-8 on Linux, so your locale should use that as well. If you work remotely on a server that sets LANG to “C” by default, it won’t work. Try setting LC_ALL=en_US.UTF-8, because that locale is installed almost everywhere by default. LC_ALL takes precedence over LANG. Have a look at the example script below to see how to make the setting automatically.
If you have a file with bogus characters in its name, even changing the locale won’t help. You have to rename the file to something sensible.
dotnet
Avdump2 also works on Linux via dotnet; it is about 30% faster than running it via mono, but appears to work with ASCII filenames only.
Create a AVDump2CL.runtimeconfig.json in the folder that contains AVDump2CL.exe with the following content:
{ "runtimeOptions": { "configProperties": { "System.GC.Server": true, "System.GC.Concurrent": true, "System.Threading.ThreadPool.MinThreads": 6, "System.Threading.ThreadPool.MaxThreads": 24 }, "tfm": "netcoreapp3.1", "framework": { "name": "Microsoft.NETCore.App", "version": "3.1.1" }, "applyPatches": true, "rollForwardOnNoCandidateFx": 1 } }
Move the Linux lib to the DLL filename so dotnet finds it
mv MediaInfo_x64.dll MediaInfo_x64.dll.windows cp -a libMediaInfo_x64.so MediaInfo_x64.dll
Workarounds
The cursor caret is gone and output resumes at the far right edge of the terminal window when I abort Avdump2 with ^C or it exits with some error.
Avdump2 doesn’t reset the console properly when you abort it or when it runs into a Mono-specific error. You can restore the cursor with setterm -cursor on and a simple echo if you write a wrapper for that. See examples below.
Can’t write log files to relative paths
Donelogs and the other options that create or update text files assume relative paths to be relative to Avdump2’s home directory. If you installed Avdump2 outside your home directory you most likely don’t have write permission there and writing the log will fail. With the --UseCWD option you can make relative paths relative to the current working directory. Absolute paths are also an option to place the logs where you have write permissions.
Wrapper Script
To make using Avdump2 more comfortable, you can use or modify this wrapper script to handle the workarounds. Save it as /usr/local/bin/avdump2cl, for example. This example assumes that you extracted Avdump2 to /usr/local/lib/avdump2.
A simple wrapper to set the LD_LIBRARY_PATH, check the locale for UTF-8 and to fix the cursor after using ^C:
#!/bin/bash AVDUMP2_HOME=/usr/local/lib/avdump2 export LD_LIBRARY_PATH=${AVDUMP2_HOME} # fix bug with ncurses 6.1+ # mono/issues/6752 export TERM=xterm # If libMediaInfo still can't be found, uncomment this. #export MONO_PATH=${AVDUMP2_HOME} # Make sure that the locale uses UTF-8 encoding. # Remove this if your file system actually uses something else. shopt -s extglob if [ -z "$LC_ALL" ]; then LC_ALL="$LANG" fi if [ "$LC_ALL" == "${LC_ALL%[Uu][Tt][Ff]?(-)8}" ]; then LC_ALL=en_US.UTF-8 fi export LC_ALL export LANG="$LC_ALL" if ! mono ${AVDUMP2_HOME}/AVDump2CL.exe "$@" ; then echo setterm -cursor on fi
Making Your Life Easier
When you made sure everything works fine, if you just want to easily hash files and generate ED2K links for adding new files, you could use this roughly-made Python 2 script. (best paired with Nautilus or Nemo actions)
Please note that this script relies on Zenity extension to display the dialog containing the generated ED2K links. If you get an error running this script, please make sure you have it installed.
#!/usr/bin/python import sys, os, subprocess ### EDIT THE FOLLOWING ACCORDING TO YOUR INFO AVDUMP2CL_PATH = "/path/to/your/AVDump2CL.exe" USERNAME = " " API_KEY = " " ### command = "mono %s --Auth=%s:%s --PrintEd2kLink -q" % (AVDUMP2CL_PATH, USERNAME, API_KEY) for arg in sys.argv[1:]: command += " \"%s\"" % arg output = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) output = output.stdout.readlines() ed2k = "" for line in output: if line[0:4] == "ed2k": ed2k += line os.system("zenity --info --text '%s'" % ed2k.rstrip())
Things That Just Don’t Work
Some things just don’t work on Linux and you’ll have to live with that. :P
Avdump2 corrupts the terminal or crashes with a DisplayBuffer error when I resize the terminal window.
The output routine used to draw the Avdump2 progress bars is not very compatible with Mono’s absolutely positioned terminal output. Enlarging the terminal window usually results in writing over previous lines and shrinking the terminal window usually results in crashes because Avdump2 tries to write to areas that don’t exist anymore.
Solutions: Don’t resize the terminal window while Avdump2 is running or use the -q option to disable the progress bars.
Inconsistent exit codes
Avdump2 exits with a return code of 0 on some errors, so don’t rely on the return code. It exits with a non-zero code on most serious errors, though.