Everything about Recurrent Neural Network Grammars Compilation (as Non-root User)

RNNG Structure in the paper

I tried to play with RNNG but encountered a lot of compilation issues (mostly due to Boost). To save others’ time, I summarize detailed compilation dependencies and solutions to common errors in this blog.

Dependencies

• Eigen (latest development version)
• CMake (latest version)
• EVALB (latest version)
• Boost

For most dependencies above, it should be easy to compile and install from source as a non-root user without further configuration.

The most tricky part is Boost. To run the RNNG code, any boost version ≥ 1.53.0 should work. However, if you want to use the pre-trained model provided by the author, Boost version has to be 1.60.0.

Compiling Boost as Non-root User

To compile Boost from source and install it to your local directory, first download the source code here. Then

tar xvf boost_1_60_0.tar.gz
cd boost_1_60_0
./bootstrap.sh --prefix=DIR_TO_INSTALL_BOOST
/b2 -j2 -q install

Note that Boost must be compiled with bzip, otherwise there will be linking error when compiling RNNG. By default, configuration of compiling with bzip is on, as long as bzip2-devel is available on the system, there should be no problem. To verify the correct compilation, do

ldd /path/to/boost-1.60.0/lib/libboost_iostreams.so

Correct compilation should yield:

…
 libbz2.so.1 => /usr/lib64/libbz2.so.1 (0x00007f458dda2000)
 libz.so.1 => /usr/lib64/libz.so.1 (0x00007f458db8c000)
…

As mentioned by the authors,

please contact us to get access to the oracle due to PTB licensing issues.

You should get the training oracle files from them to load the pre-trained model.

RNNG Configuration

The major configuration file is CMakeLists.txt. To use the local boost and Eigen, add

list(APPEND CMAKE_PREFIX_PATH "/path/to/boost-1.60.0")
list(APPEND CMAKE_INCLUDE_PATH "/path/to/eigen/include")

to the file. Create a build directory

mkdir build && cd build

Then create the Makefile by running

cmake ..

Correct configuration should yield

-- The C compiler identification is GNU 4.8.5
-- The CXX compiler identification is GNU 4.8.5
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Boost version: 1.60.0
-- Found the following Boost libraries:
--   program_options
--   iostreams
--   serialization
-- Found Eigen3: /path/to/eigen/include/eigen3 (Required is at least version "2.91.0")
-- Configuring done
-- Generating done
-- Build files have been written to: ...

Then make should succeed.

Common Compilation Errors

1. Failure at linking stage with undefined reference to iostreams:

[ 92%] Linking CXX executable nt-parser
CMakeFiles/nt-parser.dir/oracle.cc.o: In function `boost::iostreams::detail::bzip2_decompressor_impl<std::allocator<char> >::filter(char const*&, char const*, char*&, char*, bool)':
/home/sonalgupta/external/opt/boost/include/boost/iostreams/filter/bzip2.hpp:358: undefined reference to `boost::iostreams::detail::bzip2_base::before(char const*&, char const*, char*&, char*)'
CMakeFiles/nt-parser.dir/oracle.cc.o: In function `boost::iostreams::detail::bzip2_decompressor_impl<std::allocator<char> >::filter(char const*&, char const*, char*&, char*, bool) [clone .constprop.343]':
oracle.cc:(.text+0x567): undefined reference to `boost::iostreams::detail::bzip2_base::decompress()'
oracle.cc:(.text+0x56d): undefined reference to `boost::iostreams::bzip2::ok'
oracle.cc:(.text+0x57f): undefined reference to `boost::iostreams::detail::bzip2_base::after(char const*&, char*&)'
oracle.cc:(.text+0x586): undefined reference to `boost::iostreams::bzip2_error::check(int)'
oracle.cc:(.text+0x58c): undefined reference to `boost::iostreams::bzip2::stream_end'
oracle.cc:(.text+0x5b5): undefined reference to `boost::iostreams::detail::bzip2_base::do_init(bool, void* (*)(void*, int, int), void (*)(void*, void*), void*)'
oracle.cc:(.text+0x5cc): undefined reference to `boost::iostreams::detail::bzip2_base::check_end(char const*, char const*)'
CMakeFiles/nt-parser.dir/oracle.cc.o: In function `boost::detail::sp_counted_impl_p<boost::iostreams::symmetric_filter<boost::iostreams::detail::bzip2_decompressor_impl<std::allocator<char> >, std::allocator<char> >::impl>::dispose()':
oracle.cc:(.text._ZN5boost6detail17sp_counted_impl_pINS_9iostreams16symmetric_filterINS2_6detail23bzip2_decompressor_implISaIcEEES6_E4implEE7disposeEv[_ZN5boost6detail17sp_counted_impl_pINS_9iostreams16symmetric_filterINS2_6detail23bzip2_decompressor_implISaIcEEES6_E4implEE7disposeEv]+0x1c): undefined reference to `boost::iostreams::detail::bzip2_base::~bzip2_base()'
CMakeFiles/nt-parser.dir/oracle.cc.o: In function `boost::iostreams::detail::bzip2_decompressor_impl<std::allocator<char> >::close()':
oracle.cc:(.text._ZN5boost9iostreams6detail23bzip2_decompressor_implISaIcEE5closeEv[_ZN5boost9iostreams6detail23bzip2_decompressor_implISaIcEE5closeEv]+0x7): undefined reference to `boost::iostreams::detail::bzip2_base::end(bool)'
CMakeFiles/nt-parser.dir/oracle.cc.o: In function `long boost::iostreams::symmetric_filter<boost::iostreams::detail::bzip2_decompressor_impl<std::allocator<char> >, std::allocator<char> >::write<boost::iostreams::detail::linked_streambuf<char, std::char_traits<char> > >(boost::iostreams::detail::linked_streambuf<char, std::char_traits<char> >&, char const*, long)':

As noted in this post, the error is due to the missing of bzip2. The problem is that Boost must be compiled with bzip2, which is a dependency for the RNNG. The easy fix is to install boost from package manager (if you have the privilege):

yum install boost-devel (on CentOs)
or
brew install boost (on MacOS)

If you don’t have the privilege, install bzip2-devel on the system first. Boost by default is configured to find bzip2. So when configuring Boost with

./bootstrap.sh

You should be able to see bzip2 checked after successful installation of bzip2-devel.

2. Failure at running the executable binary

./nt-parser: error while loading shared libraries: libboost_program_options.so.1.60.0: cannot open shared object file: No such file or directory

The error is probably due to the missing of LD_LIBRARY_PATH, and the binary is unable to find the correct path for dynamic library. Run

export LD_LIBRARY_PATH=/path/to/boost-1.60.0/lib