Avdump2 on Linux: Difference between revisions

From AniDB
Jump to navigation Jump to search
m (Reworded the libMediaInfo requirements)
({{Avdump-current-version}})
 
(20 intermediate revisions by 6 users not shown)
Line 1: Line 1:
Avdump2 was developed on Windows, for Windows. Thanks to Mono it’s possible to run Avdump2 on Linux and all efforts are made to support all basic features on Linux as well, but handling every peculiarity of every Mono version on every Linux installation is beyond the scope of Avdump2 at this time. Therefore, some things just don’t work on Linux, or at least not right away, without some workarounds.
{{TOCright}}
{{eyecatch|WARNING|This article was written for [[Avdump2]]. The current latest version is {{Avdump-current-version}}, and the steps in this article have not yet been explicitly verified against {{Avdump-current-version}}.}}
[[Avdump2]] was developed on Windows, for Windows. Thanks to Mono it’s possible to run Avdump2 on Linux and all efforts are made to support all features on Linux as well, but handling every peculiarity of every Mono version on every Linux installation is beyond the scope of Avdump2 at this time. Therefore, some things just don’t work on Linux, or at least not right away, without some workarounds.


This page is a collection of various workarounds and suggestions that should help you get started with Avdump2 on Linux. If you need more assistance, try to catch [[User:catweax|CatWEax]] on IRC.
This page is a collection of various workarounds and suggestions that should help you get started with Avdump2 on Linux. If you need more assistance, try to catch [[User:catweax|CatWEax]] on IRC.
Line 7: Line 9:


=== Mono ===
=== Mono ===
Avdump2 has successfully been tested with Mono 2.4 on Ubuntu and with Mono 2.8 on Gentoo.
Avdump2 has successfully been tested with Mono 2.4 on Ubuntu, with Mono 2.8 on Gentoo, and Mono 5.20 on openSUSE Tumbleweed.


'''<tt>Avdump2CL.exe</tt> looks like a Windows .exe, how do I start it?'''
'''<tt>Avdump2CL.exe</tt> looks like a Windows .exe, how do I start it?'''
Line 13: Line 15:
Try making the file executable. If that doesn’t work, call it using the <tt>mono</tt> Interpreter: <tt>mono Avdump2CL.exe</tt>
Try making the file executable. If that doesn’t work, call it using the <tt>mono</tt> Interpreter: <tt>mono Avdump2CL.exe</tt>


'''Avdump2 keeps looking for <tt>/some/directory/mono</tt>'''
'''OutOfMemory exceptions'''


If you extracted Avdump2 into <tt>/some/directory</tt> and it won’t work because it can’t find <tt>/some/directory/mono</tt>, you might have a buggy Mono version. This has been observed with Mono 2.10 on Gentoo and downgrading to Mono 2.8 solved the issue. It might also work if you create a symlink from <tt>mono</tt> to <tt>Avdump2CL.exe</tt> in that directory.
On 32 bit systems Avdump2 sometimes dies with that error when processing many files at once due to heap memory fragmentation. If you use a donelog you can just restart Avdump2 and it’ll continue from where it crashed.
 
'''Mono complains about not being able to load type 'AVDump2CL.CL''''
 
The actual error message probably looks similar to this:
 
<tt>Unhandled Exception: System.TypeLoadException: Could not load type 'AVDump2CL.CL' from assembly 'AVDump2CL, Version=0.1.6438.0, Culture=neutral, PublicKeyToken=null'.</tt>
 
If you get this error then you probably don’t have all necessary mono packages installed. On a Debian-based system make sure that you have the packages <tt>libmono-corlib2.0-cil</tt> and <tt>libmono-system2.0-cil</tt> installed.


=== libMediaInfo ===
=== libMediaInfo ===
Avdump2 only works with the libMediaInfo it comes with. If it can’t find that you will usually get ''Media Info Library not found.'' It might start with a different version from elsewhere on your system, but then you’ll run into odd errors when you try to dump a file.
Avdump2 only works with the libMediaInfo it comes with. If it can’t find that you will usually get ''MediaInfoLib is missing''. It might start with a different version from elsewhere on your system, but then you’ll run into odd errors when you try to dump a file.


There are many possible causes if the library can’t be found.
There are many possible causes if the library can’t be found.
Line 24: Line 34:
'''Library search path missing'''
'''Library search path missing'''


Most likely Mono can’t find the binary library even if it’s in the same directory as <tt>Avdump2CL.exe</tt>. Then you’ll have to set the environment variable <tt>LD_LIBRARY_PATH</tt> to point to the diretory into which you extracted Avdump2. Write yourself a wrapper script to do that or have a look at the example wrapper scripts below. Sometimes you’ll also have to set <tt>MONO_PATH</tt> to point to the same directory.
Most likely Mono can’t find the binary library even if it’s in the same directory as <tt>Avdump2CL.exe</tt>. Then you’ll have to set the environment variable <tt>LD_LIBRARY_PATH</tt> to point to the directory into which you extracted Avdump2. Write yourself a wrapper script to do that or have a look at the example wrapper scripts below. Sometimes you’ll also have to set <tt>MONO_PATH</tt> to point to the same directory.


'''Incompatible GLIBC'''
'''Incompatible GLIBC'''
Line 32: Line 42:
'''Hashing works but I can’t dump anything'''
'''Hashing works but I can’t dump anything'''


If you modified ''any'' of the required files Avdump2 came with, you will usually get the misleading error ''Either the client is outdated or your username/password combination is wrong.'' So no, you can’t edit the config file either.
If you modified ''any'' of the required files Avdump2 came with, you will either get a version error or several timeouts and no meaningful error message. So no, you can’t edit the config file either.
 
'''Failed to create MediaInfoProvider'''
 
<tt>Failed to create MediaInfoProvider</tt>
 
<tt>Invalid value returned by mil. This error may indicate that your LANG or LC_ALL environment variables are not set to use UTF-8 encoding.</tt>
 
This error appears if you try to process a file with special characters in its name and your locale settings don’t match the file system encoding. Nowadays almost all file names are encoded with UTF-8 on Linux, so your locale should use that as well. If you work remotely on a server that sets LANG to “C” by default, it won’t work. Try setting LC_ALL=en_US.UTF-8, because that locale is installed almost everywhere by default. LC_ALL takes precedence over LANG. Have a look at the example script below to see how to make the setting automatically.
 
If you have a file with bogus characters in its name, even changing the locale won’t help. You have to rename the file to something sensible.
 
=== dotnet ===
Avdump2 also works on Linux via '''dotnet'''; it is about 30% faster than running it via '''mono''', but appears to work with ASCII filenames only.
 
Create a <tt>AVDump2CL.runtimeconfig.json</tt> in the folder that contains '''AVDump2CL.exe''' with the following content:
<pre>{
    "runtimeOptions": {
        "configProperties": {
            "System.GC.Server": true,
            "System.GC.Concurrent": true,
            "System.Threading.ThreadPool.MinThreads": 6,
            "System.Threading.ThreadPool.MaxThreads": 24
        },
        "tfm": "netcoreapp3.1",
        "framework": {
            "name": "Microsoft.NETCore.App",
            "version": "3.1.1"                                                                                                                                                                                                                                                                       
        },
        "applyPatches": true,
        "rollForwardOnNoCandidateFx": 1
    }
}</pre>
Move the Linux lib to the DLL filename so '''dotnet''' finds it
<pre>mv MediaInfo_x64.dll MediaInfo_x64.dll.windows
cp -a libMediaInfo_x64.so MediaInfo_x64.dll
</pre>


== Workarounds ==
== Workarounds ==
''' The cursor caret is gone and output resumes at the far right edge of the terminal window when I abort Avdump2 with ^C or it exits with some error.'''
'''The cursor caret is gone and output resumes at the far right edge of the terminal window when I abort Avdump2 with ^C or it exits with some error.'''


Avdump2 doesn’t reset the console properly when you abort it or when it runs into a Mono-specific error. You can restore the cursor with <tt>setterm -cursor on</tt> and a simple <tt>echo</tt> if you write a wrapper for that. See examples below.
Avdump2 doesn’t reset the console properly when you abort it or when it runs into a Mono-specific error. You can restore the cursor with <tt>setterm -cursor on</tt> and a simple <tt>echo</tt> if you write a wrapper for that. See examples below.


'''I can’t dump files that have spaces in their name, even if I put them in quotation marks.'''
'''Can’t write log files to relative paths'''


This has been observed with Mono 2.4 but not with Mono 2.8. If it applies to you, you can add additional quotation marks to the parameters because Avdump2 can do its own quotation handling: <tt>Avdump2CL.exe -0 "\"file with spaces.mkv\""</tt> As you can’t use standard shell expansions like <tt>*.mkv</tt> with that, you’d better use the appropriate wrapper sript from below.
Donelogs and the other options that create or update text files assume relative paths to be relative to Avdump2’s home directory. If you installed Avdump2 outside your home directory you most likely don’t have write permission there and writing the log will fail. With the <tt>--UseCWD</tt> option you can make relative paths relative to the current working directory. Absolute paths are also an option to place the logs where you have write permissions.


== Wrapper Scripts ==
== Wrapper Script ==
To make using Avdump2 more comfortable, you can use or modify one of these wrapper scripts to handle all the workarounds. Save it as <tt>/usr/local/bin/avdump2cl</tt>, for example. These examples assume that you extracted Avdump2 to <tt>/usr/local/lib/avdump2</tt>.
To make using Avdump2 more comfortable, you can use or modify this wrapper script to handle the workarounds. Save it as <tt>/usr/local/bin/avdump2cl</tt>, for example. This example assumes that you extracted Avdump2 to <tt>/usr/local/lib/avdump2</tt>.


A simple wrapper to set the <tt>LD_LIBRARY_PATH</tt> and to fix the cursor after using ^C:
A simple wrapper to set the <tt>LD_LIBRARY_PATH</tt>, check the locale for UTF-8 and to fix the cursor after using ^C:
<pre>
<pre>
#!/bin/bash
#!/bin/bash


export LD_LIBRARY_PATH=/usr/local/lib/avdump2
AVDUMP2_HOME=/usr/local/lib/avdump2
 
export LD_LIBRARY_PATH=${AVDUMP2_HOME}
 
# fix bug with ncurses 6.1+
# mono/issues/6752
export TERM=xterm


# If libMediaInfo still can't be found, uncomment this.
# If libMediaInfo still can't be found, uncomment this.
#export MONO_PATH=$LD_LIBRARY_PATH
#export MONO_PATH=${AVDUMP2_HOME}
 
# Make sure that the locale uses UTF-8 encoding.
# Remove this if your file system actually uses something else.
shopt -s extglob
if [ -z "$LC_ALL" ]; then
  LC_ALL="$LANG"
fi
if [ "$LC_ALL" == "${LC_ALL%[Uu][Tt][Ff]?(-)8}" ]; then
  LC_ALL=en_US.UTF-8
fi
export LC_ALL
export LANG="$LC_ALL"


if ! mono /usr/local/lib/avdump2/AVDump2CL.exe "$@" ; then
if ! mono ${AVDUMP2_HOME}/AVDump2CL.exe "$@" ; then
   echo
   echo
   setterm -cursor on
   setterm -cursor on
Line 61: Line 125:
</pre>
</pre>


A more elaborate wrapper to add quotation marks to filenames with spaces in them if your Mono version doesn’t handle them properly:
== Making Your Life Easier ==
When you made sure everything works fine, if you just want to easily hash files and generate ED2K links for adding new files, you could use this roughly-made Python 2 script. (best paired with Nautilus or Nemo actions)
 
Please note that this script relies on Zenity extension to display the dialog containing the generated ED2K links. If you get an error running this script, please make sure you have it installed.
 
<pre>
<pre>
#!/bin/bash
#!/usr/bin/python
 
import sys, os, subprocess
 
### EDIT THE FOLLOWING ACCORDING TO YOUR INFO
AVDUMP2CL_PATH = "/path/to/your/AVDump2CL.exe"
USERNAME = " "
API_KEY = " "
###


export LD_LIBRARY_PATH=/usr/local/lib/avdump2
command = "mono %s --Auth=%s:%s --PrintEd2kLink -q" % (AVDUMP2CL_PATH, USERNAME, API_KEY)
for arg in sys.argv[1:]:
    command += " \"%s\"" % arg


declare -i i=0
output = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
declare -a params
output = output.stdout.readlines()


while [ $# -gt 0 ]; do
ed2k = ""
  if [ -z "${1##* *}" ]; then
for line in output:
    params[$i]="\"${1}\""
     if line[0:4] == "ed2k":
  else
        ed2k += line
     params[$i]="${1}"
  fi
  shift
  i=$i+1
done


if ! mono /usr/local/lib/avdump2/AVDump2CL.exe "${params[@]}" ; then
os.system("zenity --info --text '%s'" % ed2k.rstrip())
  echo
  setterm -cursor on
fi
</pre>
</pre>


Line 91: Line 161:
'''Avdump2 corrupts the terminal or crashes with a ''DisplayBuffer'' error when I resize the terminal window.'''
'''Avdump2 corrupts the terminal or crashes with a ''DisplayBuffer'' error when I resize the terminal window.'''


The output routine used to draw the Avdump2 progress bars is not very compatible with the <tt>curses</tt>-based output used by Mono. Enlarging the terminal window usually results in writing over previous lines and shrinking the terminal window usually results in crashes because Avdump2 tries to write to areas that don’t exist anymore.
The output routine used to draw the Avdump2 progress bars is not very compatible with Mono’s absolutely positioned terminal output. Enlarging the terminal window usually results in writing over previous lines and shrinking the terminal window usually results in crashes because Avdump2 tries to write to areas that don’t exist anymore.
 
Solutions: Don’t resize the terminal window while Avdump2 is running or use the <tt>-q</tt> option to disable the progress bars.
 
'''Inconsistent exit codes'''
 
Avdump2 exits with a return code of 0 on some errors, so don’t rely on the return code. It exits with a non-zero code on most serious errors, though.


Solutions: Don’t resize the terminal window while Avdump2 is running or use the <tt>-w</tt> option to disable the progress bars.


== Unconfirmed Trouble ==
[[Category:Features]]
* Donefiles might not work with relative paths, so use an absolute path instead.
[[Category:AniDB Clients]]
* Automatically deleting files might not work, files just stay there. Delete them manually instead.
[[Category:Avdump]]
* Avdump2 might exit with a return code of <tt>0</tt> on some errors, so don’t rely on the return code.
[[Category:Development]]

Latest revision as of 01:25, 19 December 2021

WARNING This article was written for Avdump2. The current latest version is Avdump3, and the steps in this article have not yet been explicitly verified against Avdump3.

Avdump2 was developed on Windows, for Windows. Thanks to Mono it’s possible to run Avdump2 on Linux and all efforts are made to support all features on Linux as well, but handling every peculiarity of every Mono version on every Linux installation is beyond the scope of Avdump2 at this time. Therefore, some things just don’t work on Linux, or at least not right away, without some workarounds.

This page is a collection of various workarounds and suggestions that should help you get started with Avdump2 on Linux. If you need more assistance, try to catch CatWEax on IRC.

The Basics

Can't get Avdump2 to work at all? Here are some basic suggestions to set up Adump2.

Mono

Avdump2 has successfully been tested with Mono 2.4 on Ubuntu, with Mono 2.8 on Gentoo, and Mono 5.20 on openSUSE Tumbleweed.

Avdump2CL.exe looks like a Windows .exe, how do I start it?

Try making the file executable. If that doesn’t work, call it using the mono Interpreter: mono Avdump2CL.exe

OutOfMemory exceptions

On 32 bit systems Avdump2 sometimes dies with that error when processing many files at once due to heap memory fragmentation. If you use a donelog you can just restart Avdump2 and it’ll continue from where it crashed.

Mono complains about not being able to load type 'AVDump2CL.CL'

The actual error message probably looks similar to this:

Unhandled Exception: System.TypeLoadException: Could not load type 'AVDump2CL.CL' from assembly 'AVDump2CL, Version=0.1.6438.0, Culture=neutral, PublicKeyToken=null'.

If you get this error then you probably don’t have all necessary mono packages installed. On a Debian-based system make sure that you have the packages libmono-corlib2.0-cil and libmono-system2.0-cil installed.

libMediaInfo

Avdump2 only works with the libMediaInfo it comes with. If it can’t find that you will usually get MediaInfoLib is missing. It might start with a different version from elsewhere on your system, but then you’ll run into odd errors when you try to dump a file.

There are many possible causes if the library can’t be found.

Library search path missing

Most likely Mono can’t find the binary library even if it’s in the same directory as Avdump2CL.exe. Then you’ll have to set the environment variable LD_LIBRARY_PATH to point to the directory into which you extracted Avdump2. Write yourself a wrapper script to do that or have a look at the example wrapper scripts below. Sometimes you’ll also have to set MONO_PATH to point to the same directory.

Incompatible GLIBC

If your Linux installation is very old or you’re using a distribution that ships with an outdated GLIBC (CentOS is a prime offender here), Mono can’t link against libMediaInfo and Avdump2 will still complain about missing the library. To check if that’s the case, use ldd libMediaInfo*.so. If you get a message like version 'GLIBCXX_3.4.9' not found, you’re out of luck. You can either try to update GLIBC, which usually involves pain, or upgrade or change your distribution to one with a newer GLIBC.

Hashing works but I can’t dump anything

If you modified any of the required files Avdump2 came with, you will either get a version error or several timeouts and no meaningful error message. So no, you can’t edit the config file either.

Failed to create MediaInfoProvider

Failed to create MediaInfoProvider

Invalid value returned by mil. This error may indicate that your LANG or LC_ALL environment variables are not set to use UTF-8 encoding.

This error appears if you try to process a file with special characters in its name and your locale settings don’t match the file system encoding. Nowadays almost all file names are encoded with UTF-8 on Linux, so your locale should use that as well. If you work remotely on a server that sets LANG to “C” by default, it won’t work. Try setting LC_ALL=en_US.UTF-8, because that locale is installed almost everywhere by default. LC_ALL takes precedence over LANG. Have a look at the example script below to see how to make the setting automatically.

If you have a file with bogus characters in its name, even changing the locale won’t help. You have to rename the file to something sensible.

dotnet

Avdump2 also works on Linux via dotnet; it is about 30% faster than running it via mono, but appears to work with ASCII filenames only.

Create a AVDump2CL.runtimeconfig.json in the folder that contains AVDump2CL.exe with the following content:

{
    "runtimeOptions": {
        "configProperties": {
            "System.GC.Server": true,
            "System.GC.Concurrent": true,
            "System.Threading.ThreadPool.MinThreads": 6,
            "System.Threading.ThreadPool.MaxThreads": 24
        },
        "tfm": "netcoreapp3.1",
        "framework": {
            "name": "Microsoft.NETCore.App",
            "version": "3.1.1"                                                                                                                                                                                                                                                                         
        },
        "applyPatches": true,
        "rollForwardOnNoCandidateFx": 1
    }
}

Move the Linux lib to the DLL filename so dotnet finds it

mv MediaInfo_x64.dll MediaInfo_x64.dll.windows
cp -a libMediaInfo_x64.so MediaInfo_x64.dll

Workarounds

The cursor caret is gone and output resumes at the far right edge of the terminal window when I abort Avdump2 with ^C or it exits with some error.

Avdump2 doesn’t reset the console properly when you abort it or when it runs into a Mono-specific error. You can restore the cursor with setterm -cursor on and a simple echo if you write a wrapper for that. See examples below.

Can’t write log files to relative paths

Donelogs and the other options that create or update text files assume relative paths to be relative to Avdump2’s home directory. If you installed Avdump2 outside your home directory you most likely don’t have write permission there and writing the log will fail. With the --UseCWD option you can make relative paths relative to the current working directory. Absolute paths are also an option to place the logs where you have write permissions.

Wrapper Script

To make using Avdump2 more comfortable, you can use or modify this wrapper script to handle the workarounds. Save it as /usr/local/bin/avdump2cl, for example. This example assumes that you extracted Avdump2 to /usr/local/lib/avdump2.

A simple wrapper to set the LD_LIBRARY_PATH, check the locale for UTF-8 and to fix the cursor after using ^C:

#!/bin/bash

AVDUMP2_HOME=/usr/local/lib/avdump2

export LD_LIBRARY_PATH=${AVDUMP2_HOME}

# fix bug with ncurses 6.1+ 
# mono/issues/6752
export TERM=xterm

# If libMediaInfo still can't be found, uncomment this.
#export MONO_PATH=${AVDUMP2_HOME}

# Make sure that the locale uses UTF-8 encoding.
# Remove this if your file system actually uses something else.
shopt -s extglob
if [ -z "$LC_ALL" ]; then
  LC_ALL="$LANG"
fi
if [ "$LC_ALL" == "${LC_ALL%[Uu][Tt][Ff]?(-)8}" ]; then
  LC_ALL=en_US.UTF-8
fi
export LC_ALL
export LANG="$LC_ALL"

if ! mono ${AVDUMP2_HOME}/AVDump2CL.exe "$@" ; then
  echo
  setterm -cursor on
fi

Making Your Life Easier

When you made sure everything works fine, if you just want to easily hash files and generate ED2K links for adding new files, you could use this roughly-made Python 2 script. (best paired with Nautilus or Nemo actions)

Please note that this script relies on Zenity extension to display the dialog containing the generated ED2K links. If you get an error running this script, please make sure you have it installed.

#!/usr/bin/python

import sys, os, subprocess

### EDIT THE FOLLOWING ACCORDING TO YOUR INFO
AVDUMP2CL_PATH = "/path/to/your/AVDump2CL.exe"
USERNAME = " "
API_KEY = " "
### 

command = "mono %s --Auth=%s:%s --PrintEd2kLink -q" % (AVDUMP2CL_PATH, USERNAME, API_KEY)
for arg in sys.argv[1:]:
    command += " \"%s\"" % arg

output = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = output.stdout.readlines()

ed2k = ""
for line in output:
    if line[0:4] == "ed2k":
        ed2k += line

os.system("zenity --info --text '%s'" % ed2k.rstrip())

Things That Just Don’t Work

Some things just don’t work on Linux and you’ll have to live with that. :P

Avdump2 corrupts the terminal or crashes with a DisplayBuffer error when I resize the terminal window.

The output routine used to draw the Avdump2 progress bars is not very compatible with Mono’s absolutely positioned terminal output. Enlarging the terminal window usually results in writing over previous lines and shrinking the terminal window usually results in crashes because Avdump2 tries to write to areas that don’t exist anymore.

Solutions: Don’t resize the terminal window while Avdump2 is running or use the -q option to disable the progress bars.

Inconsistent exit codes

Avdump2 exits with a return code of 0 on some errors, so don’t rely on the return code. It exits with a non-zero code on most serious errors, though.