Avdump2 on Linux: Difference between revisions

From AniDB
Jump to navigation Jump to search
m (Updated soon to be obsolete workaround.)
(11 intermediate revisions by 6 users not shown)
Line 1: Line 1:
Avdump2 was developed on Windows, for Windows. Thanks to Mono it’s possible to run Avdump2 on Linux and all efforts are made to support all basic features on Linux as well, but handling every peculiarity of every Mono version on every Linux installation is beyond the scope of Avdump2 at this time. Therefore, some things just don’t work on Linux, or at least not right away, without some workarounds.
{{TOCright}}
 
[[Avdump2]] was developed on Windows, for Windows. Thanks to Mono it’s possible to run Avdump2 on Linux and all efforts are made to support all features on Linux as well, but handling every peculiarity of every Mono version on every Linux installation is beyond the scope of Avdump2 at this time. Therefore, some things just don’t work on Linux, or at least not right away, without some workarounds.


This page is a collection of various workarounds and suggestions that should help you get started with Avdump2 on Linux. If you need more assistance, try to catch [[User:catweax|CatWEax]] on IRC.
This page is a collection of various workarounds and suggestions that should help you get started with Avdump2 on Linux. If you need more assistance, try to catch [[User:catweax|CatWEax]] on IRC.
Line 7: Line 9:


=== Mono ===
=== Mono ===
Avdump2 has successfully been tested with Mono 2.4 on Ubuntu and with Mono 2.8 on Gentoo.
Avdump2 has successfully been tested with Mono 2.4 on Ubuntu, with Mono 2.8 on Gentoo, and Mono 5.20 on openSUSE Tumbleweed.


'''<tt>Avdump2CL.exe</tt> looks like a Windows .exe, how do I start it?'''
'''<tt>Avdump2CL.exe</tt> looks like a Windows .exe, how do I start it?'''
Line 13: Line 15:
Try making the file executable. If that doesn’t work, call it using the <tt>mono</tt> Interpreter: <tt>mono Avdump2CL.exe</tt>
Try making the file executable. If that doesn’t work, call it using the <tt>mono</tt> Interpreter: <tt>mono Avdump2CL.exe</tt>


'''Avdump2 keeps looking for <tt>/some/directory/mono</tt>'''
'''OutOfMemory exceptions'''
 
On 32 bit systems Avdump2 sometimes dies with that error when processing many files at once due to heap memory fragmentation. If you use a donelog you can just restart Avdump2 and it’ll continue from where it crashed.
 
'''Mono complains about not being able to load type 'AVDump2CL.CL''''
 
The actual error message probably looks similar to this:
 
<tt>Unhandled Exception: System.TypeLoadException: Could not load type 'AVDump2CL.CL' from assembly 'AVDump2CL, Version=0.1.6438.0, Culture=neutral, PublicKeyToken=null'.</tt>


If you extracted Avdump2 into <tt>/some/directory</tt> and it won’t work because it can’t find <tt>/some/directory/mono</tt>, you might have a buggy Mono version. This has been observed with Mono 2.10 on Gentoo and downgrading to Mono 2.8 solved the issue. It might also work if you create a symlink from <tt>mono</tt> to <tt>Avdump2CL.exe</tt> in that directory. This only applies to versions up to 5080. Don’t use this workaround with version 5081 and newer.
If you get this error then you probably don’t have all necessary mono packages installed. On a Debian-based system make sure that you have the packages <tt>libmono-corlib2.0-cil</tt> and <tt>libmono-system2.0-cil</tt> installed.


=== libMediaInfo ===
=== libMediaInfo ===
Avdump2 only works with the libMediaInfo it comes with. If it can’t find that you will usually get ''Media Info Library not found.'' It might start with a different version from elsewhere on your system, but then you’ll run into odd errors when you try to dump a file.
Avdump2 only works with the libMediaInfo it comes with. If it can’t find that you will usually get ''MediaInfoLib is missing''. It might start with a different version from elsewhere on your system, but then you’ll run into odd errors when you try to dump a file.


There are many possible causes if the library can’t be found.
There are many possible causes if the library can’t be found.
Line 32: Line 42:
'''Hashing works but I can’t dump anything'''
'''Hashing works but I can’t dump anything'''


If you modified ''any'' of the required files Avdump2 came with, you will usually get the misleading error ''Either the client is outdated or your username/password combination is wrong.'' So no, you can’t edit the config file either.
If you modified ''any'' of the required files Avdump2 came with, you will either get a version error or several timeouts and no meaningful error message. So no, you can’t edit the config file either.
 
'''Failed to create MediaInfoProvider'''
 
<tt>Failed to create MediaInfoProvider</tt>
 
<tt>Invalid value returned by mil. This error may indicate that your LANG or LC_ALL environment variables are not set to use UTF-8 encoding.</tt>
 
This error appears if you try to process a file with special characters in its name and your locale settings don’t match the file system encoding. Nowadays almost all file names are encoded with UTF-8 on Linux, so your locale should use that as well. If you work remotely on a server that sets LANG to “C” by default, it won’t work. Try setting LC_ALL=en_US.UTF-8, because that locale is installed almost everywhere by default. LC_ALL takes precedence over LANG. Have a look at the example script below to see how to make the setting automatically.
 
If you have a file with bogus characters in its name, even changing the locale won’t help. You have to rename the file to something sensible.
 
=== dotnet ===
Avdump2 also works on Linux via '''dotnet'''; it is about 30% faster than running it via '''mono''', but appears to work with ASCII filenames only.
 
Create a <tt>AVDump2CL.runtimeconfig.json</tt> in the folder that contains '''AVDump2CL.exe''' with the following content:
<pre>{
    "runtimeOptions": {
        "configProperties": {
            "System.GC.Server": true,
            "System.GC.Concurrent": true,
            "System.Threading.ThreadPool.MinThreads": 6,
            "System.Threading.ThreadPool.MaxThreads": 24
        },
        "tfm": "netcoreapp3.1",
        "framework": {
            "name": "Microsoft.NETCore.App",
            "version": "3.1.1"                                                                                                                                                                                                                                                                       
        },
        "applyPatches": true,
        "rollForwardOnNoCandidateFx": 1
    }
}</pre>
Move the Linux lib to the DLL filename so '''dotnet''' finds it
<pre>mv MediaInfo_x64.dll MediaInfo_x64.dll.windows
cp -a libMediaInfo_x64.so MediaInfo_x64.dll
</pre>


== Workarounds ==
== Workarounds ==
''' The cursor caret is gone and output resumes at the far right edge of the terminal window when I abort Avdump2 with ^C or it exits with some error.'''
'''The cursor caret is gone and output resumes at the far right edge of the terminal window when I abort Avdump2 with ^C or it exits with some error.'''


Avdump2 doesn’t reset the console properly when you abort it or when it runs into a Mono-specific error. You can restore the cursor with <tt>setterm -cursor on</tt> and a simple <tt>echo</tt> if you write a wrapper for that. See examples below.
Avdump2 doesn’t reset the console properly when you abort it or when it runs into a Mono-specific error. You can restore the cursor with <tt>setterm -cursor on</tt> and a simple <tt>echo</tt> if you write a wrapper for that. See examples below.
'''Can’t write log files to relative paths'''
Donelogs and the other options that create or update text files assume relative paths to be relative to Avdump2’s home directory. If you installed Avdump2 outside your home directory you most likely don’t have write permission there and writing the log will fail. With the <tt>--UseCWD</tt> option you can make relative paths relative to the current working directory. Absolute paths are also an option to place the logs where you have write permissions.


== Wrapper Script ==
== Wrapper Script ==
To make using Avdump2 more comfortable, you can use or modify this wrapper script to handle the workarounds. Save it as <tt>/usr/local/bin/avdump2cl</tt>, for example. This example assumes that you extracted Avdump2 to <tt>/usr/local/lib/avdump2</tt>.
To make using Avdump2 more comfortable, you can use or modify this wrapper script to handle the workarounds. Save it as <tt>/usr/local/bin/avdump2cl</tt>, for example. This example assumes that you extracted Avdump2 to <tt>/usr/local/lib/avdump2</tt>.


A simple wrapper to set the <tt>LD_LIBRARY_PATH</tt> and to fix the cursor after using ^C:
A simple wrapper to set the <tt>LD_LIBRARY_PATH</tt>, check the locale for UTF-8 and to fix the cursor after using ^C:
<pre>
<pre>
#!/bin/bash
#!/bin/bash


export LD_LIBRARY_PATH=/usr/local/lib/avdump2
AVDUMP2_HOME=/usr/local/lib/avdump2
 
export LD_LIBRARY_PATH=${AVDUMP2_HOME}
 
# fix bug with ncurses 6.1+
# mono/issues/6752
export TERM=xterm


# If libMediaInfo still can't be found, uncomment this.
# If libMediaInfo still can't be found, uncomment this.
#export MONO_PATH=$LD_LIBRARY_PATH
#export MONO_PATH=${AVDUMP2_HOME}
 
# Make sure that the locale uses UTF-8 encoding.
# Remove this if your file system actually uses something else.
shopt -s extglob
if [ -z "$LC_ALL" ]; then
  LC_ALL="$LANG"
fi
if [ "$LC_ALL" == "${LC_ALL%[Uu][Tt][Ff]?(-)8}" ]; then
  LC_ALL=en_US.UTF-8
fi
export LC_ALL
export LANG="$LC_ALL"


if ! mono /usr/local/lib/avdump2/AVDump2CL.exe "$@" ; then
if ! mono ${AVDUMP2_HOME}/AVDump2CL.exe "$@" ; then
   echo
   echo
   setterm -cursor on
   setterm -cursor on
fi
fi
</pre>
== Making Your Life Easier ==
When you made sure everything works fine, if you just want to easily hash files and generate ED2K links for adding new files, you could use this roughly-made Python 2 script. (best paired with Nautilus or Nemo actions)
Please note that this script relies on Zenity extension to display the dialog containing the generated ED2K links. If you get an error running this script, please make sure you have it installed.
<pre>
#!/usr/bin/python
import sys, os, subprocess
### EDIT THE FOLLOWING ACCORDING TO YOUR INFO
AVDUMP2CL_PATH = "/path/to/your/AVDump2CL.exe"
USERNAME = " "
API_KEY = " "
###
command = "mono %s --Auth=%s:%s --PrintEd2kLink -q" % (AVDUMP2CL_PATH, USERNAME, API_KEY)
for arg in sys.argv[1:]:
    command += " \"%s\"" % arg
output = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = output.stdout.readlines()
ed2k = ""
for line in output:
    if line[0:4] == "ed2k":
        ed2k += line
os.system("zenity --info --text '%s'" % ed2k.rstrip())
</pre>
</pre>


Line 62: Line 161:
'''Avdump2 corrupts the terminal or crashes with a ''DisplayBuffer'' error when I resize the terminal window.'''
'''Avdump2 corrupts the terminal or crashes with a ''DisplayBuffer'' error when I resize the terminal window.'''


The output routine used to draw the Avdump2 progress bars is not very compatible with the <tt>curses</tt>-based output used by Mono. Enlarging the terminal window usually results in writing over previous lines and shrinking the terminal window usually results in crashes because Avdump2 tries to write to areas that don’t exist anymore.
The output routine used to draw the Avdump2 progress bars is not very compatible with Mono’s absolutely positioned terminal output. Enlarging the terminal window usually results in writing over previous lines and shrinking the terminal window usually results in crashes because Avdump2 tries to write to areas that don’t exist anymore.


Solutions: Don’t resize the terminal window while Avdump2 is running or use the <tt>-q</tt> option to disable the progress bars.
Solutions: Don’t resize the terminal window while Avdump2 is running or use the <tt>-q</tt> option to disable the progress bars.


== Other Known Problems ==
'''Inconsistent exit codes'''
These problems have been reported by more than one user, but we haven’t been able to track them down yet. Please contact us if you can provide more information on them.
 
* Donefiles and the other options that create or update text files assume relative paths to be relative to avdump2’s directory. If you don’t have write permission there, relative paths won’t work. Either create a subdirectory with permissions for your user or use absolute paths instead, which work just fine.


* On 32 bit Linux you might get spurious OutOfMemory exceptions, though the error message might say something about a BlockConsumer throwing an error. Usually it works fine on the second attempt, so just retry the file.
Avdump2 exits with a return code of 0 on some errors, so don’t rely on the return code. It exits with a non-zero code on most serious errors, though.


== Unconfirmed Trouble ==
These problems have been reported by only one user and may or may not be caused by an actual bug in Avdump2. We’d like to know if you got any input on any of this.


* Avdump2 might exit with a return code of 0 on some errors, so don’t rely on the return code.
[[Category:Features]]
[[Category:AniDB Clients]]
[[Category:Avdump]]
[[Category:Development]]

Revision as of 23:37, 27 January 2020

Avdump2 was developed on Windows, for Windows. Thanks to Mono it’s possible to run Avdump2 on Linux and all efforts are made to support all features on Linux as well, but handling every peculiarity of every Mono version on every Linux installation is beyond the scope of Avdump2 at this time. Therefore, some things just don’t work on Linux, or at least not right away, without some workarounds.

This page is a collection of various workarounds and suggestions that should help you get started with Avdump2 on Linux. If you need more assistance, try to catch CatWEax on IRC.

The Basics

Can't get Avdump2 to work at all? Here are some basic suggestions to set up Adump2.

Mono

Avdump2 has successfully been tested with Mono 2.4 on Ubuntu, with Mono 2.8 on Gentoo, and Mono 5.20 on openSUSE Tumbleweed.

Avdump2CL.exe looks like a Windows .exe, how do I start it?

Try making the file executable. If that doesn’t work, call it using the mono Interpreter: mono Avdump2CL.exe

OutOfMemory exceptions

On 32 bit systems Avdump2 sometimes dies with that error when processing many files at once due to heap memory fragmentation. If you use a donelog you can just restart Avdump2 and it’ll continue from where it crashed.

Mono complains about not being able to load type 'AVDump2CL.CL'

The actual error message probably looks similar to this:

Unhandled Exception: System.TypeLoadException: Could not load type 'AVDump2CL.CL' from assembly 'AVDump2CL, Version=0.1.6438.0, Culture=neutral, PublicKeyToken=null'.

If you get this error then you probably don’t have all necessary mono packages installed. On a Debian-based system make sure that you have the packages libmono-corlib2.0-cil and libmono-system2.0-cil installed.

libMediaInfo

Avdump2 only works with the libMediaInfo it comes with. If it can’t find that you will usually get MediaInfoLib is missing. It might start with a different version from elsewhere on your system, but then you’ll run into odd errors when you try to dump a file.

There are many possible causes if the library can’t be found.

Library search path missing

Most likely Mono can’t find the binary library even if it’s in the same directory as Avdump2CL.exe. Then you’ll have to set the environment variable LD_LIBRARY_PATH to point to the directory into which you extracted Avdump2. Write yourself a wrapper script to do that or have a look at the example wrapper scripts below. Sometimes you’ll also have to set MONO_PATH to point to the same directory.

Incompatible GLIBC

If your Linux installation is very old or you’re using a distribution that ships with an outdated GLIBC (CentOS is a prime offender here), Mono can’t link against libMediaInfo and Avdump2 will still complain about missing the library. To check if that’s the case, use ldd libMediaInfo*.so. If you get a message like version 'GLIBCXX_3.4.9' not found, you’re out of luck. You can either try to update GLIBC, which usually involves pain, or upgrade or change your distribution to one with a newer GLIBC.

Hashing works but I can’t dump anything

If you modified any of the required files Avdump2 came with, you will either get a version error or several timeouts and no meaningful error message. So no, you can’t edit the config file either.

Failed to create MediaInfoProvider

Failed to create MediaInfoProvider

Invalid value returned by mil. This error may indicate that your LANG or LC_ALL environment variables are not set to use UTF-8 encoding.

This error appears if you try to process a file with special characters in its name and your locale settings don’t match the file system encoding. Nowadays almost all file names are encoded with UTF-8 on Linux, so your locale should use that as well. If you work remotely on a server that sets LANG to “C” by default, it won’t work. Try setting LC_ALL=en_US.UTF-8, because that locale is installed almost everywhere by default. LC_ALL takes precedence over LANG. Have a look at the example script below to see how to make the setting automatically.

If you have a file with bogus characters in its name, even changing the locale won’t help. You have to rename the file to something sensible.

dotnet

Avdump2 also works on Linux via dotnet; it is about 30% faster than running it via mono, but appears to work with ASCII filenames only.

Create a AVDump2CL.runtimeconfig.json in the folder that contains AVDump2CL.exe with the following content:

{
    "runtimeOptions": {
        "configProperties": {
            "System.GC.Server": true,
            "System.GC.Concurrent": true,
            "System.Threading.ThreadPool.MinThreads": 6,
            "System.Threading.ThreadPool.MaxThreads": 24
        },
        "tfm": "netcoreapp3.1",
        "framework": {
            "name": "Microsoft.NETCore.App",
            "version": "3.1.1"                                                                                                                                                                                                                                                                         
        },
        "applyPatches": true,
        "rollForwardOnNoCandidateFx": 1
    }
}

Move the Linux lib to the DLL filename so dotnet finds it

mv MediaInfo_x64.dll MediaInfo_x64.dll.windows
cp -a libMediaInfo_x64.so MediaInfo_x64.dll

Workarounds

The cursor caret is gone and output resumes at the far right edge of the terminal window when I abort Avdump2 with ^C or it exits with some error.

Avdump2 doesn’t reset the console properly when you abort it or when it runs into a Mono-specific error. You can restore the cursor with setterm -cursor on and a simple echo if you write a wrapper for that. See examples below.

Can’t write log files to relative paths

Donelogs and the other options that create or update text files assume relative paths to be relative to Avdump2’s home directory. If you installed Avdump2 outside your home directory you most likely don’t have write permission there and writing the log will fail. With the --UseCWD option you can make relative paths relative to the current working directory. Absolute paths are also an option to place the logs where you have write permissions.

Wrapper Script

To make using Avdump2 more comfortable, you can use or modify this wrapper script to handle the workarounds. Save it as /usr/local/bin/avdump2cl, for example. This example assumes that you extracted Avdump2 to /usr/local/lib/avdump2.

A simple wrapper to set the LD_LIBRARY_PATH, check the locale for UTF-8 and to fix the cursor after using ^C:

#!/bin/bash

AVDUMP2_HOME=/usr/local/lib/avdump2

export LD_LIBRARY_PATH=${AVDUMP2_HOME}

# fix bug with ncurses 6.1+ 
# mono/issues/6752
export TERM=xterm

# If libMediaInfo still can't be found, uncomment this.
#export MONO_PATH=${AVDUMP2_HOME}

# Make sure that the locale uses UTF-8 encoding.
# Remove this if your file system actually uses something else.
shopt -s extglob
if [ -z "$LC_ALL" ]; then
  LC_ALL="$LANG"
fi
if [ "$LC_ALL" == "${LC_ALL%[Uu][Tt][Ff]?(-)8}" ]; then
  LC_ALL=en_US.UTF-8
fi
export LC_ALL
export LANG="$LC_ALL"

if ! mono ${AVDUMP2_HOME}/AVDump2CL.exe "$@" ; then
  echo
  setterm -cursor on
fi

Making Your Life Easier

When you made sure everything works fine, if you just want to easily hash files and generate ED2K links for adding new files, you could use this roughly-made Python 2 script. (best paired with Nautilus or Nemo actions)

Please note that this script relies on Zenity extension to display the dialog containing the generated ED2K links. If you get an error running this script, please make sure you have it installed.

#!/usr/bin/python

import sys, os, subprocess

### EDIT THE FOLLOWING ACCORDING TO YOUR INFO
AVDUMP2CL_PATH = "/path/to/your/AVDump2CL.exe"
USERNAME = " "
API_KEY = " "
### 

command = "mono %s --Auth=%s:%s --PrintEd2kLink -q" % (AVDUMP2CL_PATH, USERNAME, API_KEY)
for arg in sys.argv[1:]:
    command += " \"%s\"" % arg

output = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = output.stdout.readlines()

ed2k = ""
for line in output:
    if line[0:4] == "ed2k":
        ed2k += line

os.system("zenity --info --text '%s'" % ed2k.rstrip())

Things That Just Don’t Work

Some things just don’t work on Linux and you’ll have to live with that. :P

Avdump2 corrupts the terminal or crashes with a DisplayBuffer error when I resize the terminal window.

The output routine used to draw the Avdump2 progress bars is not very compatible with Mono’s absolutely positioned terminal output. Enlarging the terminal window usually results in writing over previous lines and shrinking the terminal window usually results in crashes because Avdump2 tries to write to areas that don’t exist anymore.

Solutions: Don’t resize the terminal window while Avdump2 is running or use the -q option to disable the progress bars.

Inconsistent exit codes

Avdump2 exits with a return code of 0 on some errors, so don’t rely on the return code. It exits with a non-zero code on most serious errors, though.