DeepVariant 安装出现的问题

我是一名 BGI 的工程师。我在安装 DeepVariant 的时候,遇到一些问题。我希望在 Tensorflow 讨论区寻找一些帮助。DeepVariant 的安装脚本需要安装 Tensorflow,CUDA,CUDNN 以及 bazel 等依赖的软件工具包。安装 DeepVariant 时,需要先运行 build-prereq.sh 脚本用于安装 DeepVariant 的配置环境,build-prereq.sh 脚本分别调用了 seeting.sh 以及 run-prereq.sh 用于设置安装包的版本以及安装 Tensorflow、CUDA 与 CUDNN 等依赖包,buid-prereq.sh 运行起来没有遇到问题。但是在运行 build_and_test.sh 测试脚本时,我遇到了一些问题。build_and_test.sh 脚本调用 bazel 对 DeepVariant 进行编译,问题出现的原因我认为就是由于 bazel 调用有问题。安装 DeepVariant 遇到的问题主要有两个:

1.我在一台远程的服务器上安装 DeepVariant 时,在该服务器中安装的 bazel 是最新的 0.12 版本,运行 buil_and_test.sh 出现如下的错误:

如果在我自己的 home 目录上调用 bazel,是可以调用的,如下图 1 所示。

但是进入 DeepVariant 的安装包路径 /home/lujianliang/deepvariant-r0.6,bazel 是找不到的,而且运行./build_and_test.sh 也是出现同样的报错,如下图所示。

2.我的同事在另外一台本地的服务器也尝试安装 DeepVariant 时,也是在运行最后的测试脚本 build_and_test.sh 遇到的问题。我们尝试 0.6 版本的 DeepVariant 与 0.5 版本的 DeepVariant,都遇到同样的 build_and_test.sh 调用 bazel 的问题。我们使用 0.6 版本的 DeepVariant 中 build_and_test.sh 测试,其中 bazel 的版本是 0.11.0,出现如下的问题:

(14:19:27) ERROR: /home/suanfa/Downloads/deepvariant-r0.6/third_party/nucleus/io/BUILD:400:1: no such package '@com_google_absl//absl/memory': The repository could not be resolved and referenced by '//third_party/nucleus/io:vcf_writer'
(14:19:27) ERROR: Analysis of target '//deepvariant/labeler:labeled_examples_to_vcf' failed; build aborted: no such package '@com_google_absl//absl/memory': The repository could not be resolved
(14:19:27) INFO: Elapsed time: 33.556s (14:19:27) FAILED: Build did NOT complete successfully (53 packages loaded)
(14:19:27) ERROR: Couldn't start the build. Unable to run tests

我们使用 0.5 版本的 DeepVariant 中 build_and_test.sh 测试,其中 bazel 的版本是 0.9.0,出现如下的问题:

(14:21:41) INFO: Current date is 2018-04-19
(14:21:44) ERROR: /home/suanfa/Downloads/deepvariant-r0.5/deepvariant/core/BUILD:126:1: no such package '@com_google_absl//absl/memory': The repository could not be resolved and referenced by '//deepvariant/core:vcf_writer'
(14:21:44) ERROR: Analysis of target '//deepvariant/core:vcf_writer' failed; build aborted: no such package '@com_google_absl//absl/memory': The repository could not be resolved
(14:21:44) INFO: Elapsed time: 3.102s
(14:21:44) FAILED: Build did NOT complete successfully (21 packages loaded)
    currently loading: @protobuf_archive// ... (3 packages)
(14:21:44) ERROR: Couldn't start the build. Unable to run tests

我之前没有用过 bazel,对 bazel 的工作原理不太熟悉。希望讨论区有人可以提供一些帮助,感谢!

build-prereq.sh、settings.sh、run-prereq.sh 以及 build_and_test.sh 的具体脚本如下:

以下是 build-prereq.sh 脚本

#!/bin/bash
set -euo pipefail

# Copyright 2017 Google Inc.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#
# 1. Redistributions of source code must retain the above copyright notice,
#    this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# 3. Neither the name of the copyright holder nor the names of its
#    contributors may be used to endorse or promote products derived from this
#    software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.

echo ========== Load config settings.

source settings.sh

################################################################################
# Misc. setup
################################################################################

note_build_stage "Install the runtime packages"

./run-prereq.sh

note_build_stage "Update package list"

sudo -H apt-get -qq -y update

note_build_stage "Install development packages"

sudo -H apt-get -y install pkg-config zip g++ zlib1g-dev unzip curl git

################################################################################
# Java
################################################################################

note_build_stage "Install Java and friends"

# Java is available on Kokoro, so we add this cutout.
if ! java -version 2>&1 | fgrep "1.8"; then
  echo "No Java 8, will install."
  sudo -H apt-get install -y software-properties-common debconf-utils
  # Debian needs authentication.
  # ([url=http://www.webupd8.org/2014/03/how-to-install-oracle-java-8-in-debian.html] http://www.webupd8.org/2014/03/how-to-install-oracle-java-8-in-debian.html [/url])
  [[ $(lsb_release -d | grep 'Debian') ]] && \
    sudo -H apt-get install -y gnupg dirmngr && \
    sudo -H apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys EEA14886
  sudo add-apt-repository -y ppa:webupd8team/java
  sudo -H apt-get -qq -y update
  echo "oracle-java8-installer shared/accepted-oracle-license-v1-1 select true" | sudo debconf-set-selections
  sudo -H apt-get install -y oracle-java8-installer
  sudo -H apt-get -y install ca-certificates-java
  sudo update-ca-certificates -f
else
  echo "Java 8 found, will not reinstall."
fi

################################################################################
# bazel
################################################################################

note_build_stage "Install bazel"

function update_bazel_linux {
  BAZEL_VERSION=$1
  rm -rf ~/bazel
  mkdir ~/bazel

  pushd ~/bazel
  curl -L -O [url=https://github.com/bazelbuild/bazel/releases/download/] https://github.com/bazelbuild/bazel/releases/download/[/url]"${BAZEL_VERSION}"/bazel-"${BAZEL_VERSION}"-installer-linux-x86_64.sh
  chmod +x bazel-*.sh
  ./bazel-"${BAZEL_VERSION}"-installer-linux-x86_64.sh --user
  rm bazel-"${BAZEL_VERSION}"-installer-linux-x86_64.sh
  popd

  PATH="$HOME/bin:$PATH"
}

bazel_ver="0.11.0"
if
  v=$(bazel --bazelrc=/dev/null --nomaster_bazelrc version) &&
  echo "$v" | awk -v b="$bazel_ver" '/Build label/ { exit ($3 != b)}'
then
  echo "Bazel $bazel_ver already installed on the machine, not reinstalling"
else
  update_bazel_linux "$bazel_ver"
fi

################################################################################
# CLIF
################################################################################

note_build_stage "Install CLIF binary"

if [[ -e /usr/local/clif/bin/pyclif ]];
then
  echo "CLIF already installed."
else
  # Figure out which linux installation we are on to fetch an appropriate
  # version of the pre-built CLIF binary. Note that we only support now Ubuntu
  # 14 and 16.
  case "$(lsb_release -d)" in
    *Ubuntu*16.*.*) export DV_PLATFORM="ubuntu-16" ;;
    *Ubuntu*14.*.*) export DV_PLATFORM="ubuntu-14" ;;
    *Debian*9.*)    export DV_PLATFORM="debian" ;;
    *Debian*rodete) export DV_PLATFORM="debian" ;;
    *) echo "CLIF is not installed on this machine and a prebuilt binary is not
unavailable for this platform. Please install CLIF at
[url=https://github.com/google/clif] https://github.com/google/clif [/url] before continuing."
    exit 1
  esac

  OSS_CLIF_CURL_ROOT="${DV_PACKAGE_CURL_PATH}/oss_clif"
  OSS_CLIF_PKG="oss_clif.${DV_PLATFORM}.latest.tgz"

  if [[ ! -f "/tmp/${OSS_CLIF_PKG}" ]]; then
    curl "${OSS_CLIF_CURL_ROOT}/${OSS_CLIF_PKG}" > /tmp/${OSS_CLIF_PKG}
  fi

  (cd / && sudo tar xzf "/tmp/${OSS_CLIF_PKG}")
  sudo ldconfig  # Reload shared libraries.
fi

################################################################################
# TensorFlow
################################################################################

note_build_stage "Download and configure TensorFlow sources"

(cd .. &&
 git clone [url=https://github.com/tensorflow/tensorflow] https://github.com/tensorflow/tensorflow [/url] &&
 cd tensorflow &&
 echo | ./configure)

note_build_stage "build-prereq.sh complete"

以下是 settings.sh 脚本

# Copyright 2017 Google Inc.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#
# 1. Redistributions of source code must retain the above copyright notice,
#    this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# 3. Neither the name of the copyright holder nor the names of its
#    contributors may be used to endorse or promote products derived from this
#    software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.

# Source this file---these options are needed for TF config and for
# successive bazel runs.

# Set this to 1 if the system image already has TensorFlow preinstalled.  This
# will skip the installation of TensorFlow.
export DV_USE_PREINSTALLED_TF="${DV_USE_PREINSTALLED_TF:-0}"

export TF_CUDA_CLANG=0
export TF_ENABLE_XLA=0
export TF_NEED_CUDA=0
export TF_NEED_GCP=1
export TF_NEED_GDR=0
export TF_NEED_HDFS=0
export TF_NEED_JEMALLOC=0
export TF_NEED_MKL=0
export TF_NEED_MPI=0
export TF_NEED_OPENCL=0
export TF_NEED_OPENCL_SYCL=0
export TF_NEED_S3=0
export TF_NEED_VERBS=0

# Used if TF_NEED_CUDA=1
export TF_CUDA_VERSION="8.0"
export CUDA_TOOLKIT_PATH="/usr/local/cuda"
export TF_CUDNN_VERSION="6"
export CUDNN_INSTALL_PATH="/usr/lib/x86_64-linux-gnu"

# Path to the public bucket containing DeepVariant-related artifacts.
export DEEPVARIANT_BUCKET="gs://deepvariant"
export DV_PACKAGE_BUCKET_PATH="${DEEPVARIANT_BUCKET}/packages"
export DV_PACKAGE_CURL_PATH="https://storage.googleapis.com/deepvariant/packages"

# Set this to 1 to use DeepVariant with GPUs. Set it to an already existing
# value in the environment (allowing command line control of the build),
# defaulting to 0 (CPU only build).
export DV_GPU_BUILD="${DV_GPU_BUILD:-0}"

# If this variable is set to 1, DeepVariant will use a TensorFlow wheel file
# compiled to use AVX and SSE instructions. This instructions require Sandy
# Bridge or better chipsets on the host machine. The default TensorFlow wheel
# files don't contain these instructions (and thereby run on a broader set of
# CPUs). Using this optimized wheel reduces the runtime of DeepVariant's
# call_variants step by ~20%. This is called the GCP (Google Cloud Platform)
# optimized wheel because all GCP instances have at least Sandy Bridge or better
# chipsets, so this wheel should run anywhere on GCP.
export DV_USE_GCP_OPTIMIZED_TF_WHL="${DV_USE_GCP_OPTIMIZED_TF_WHL:-1}"
export GCP_OPTIMIZED_TF_WHL_FILENAME="tensorflow-1.4.1.deepvariant_gcp-cp27-none-linux_x86_64.whl"
export GCP_OPTIMIZED_TF_WHL_PATH="${DV_PACKAGE_BUCKET_PATH}/tensorflow"
export GCP_OPTIMIZED_TF_WHL_CURL_PATH="${DV_PACKAGE_CURL_PATH}/tensorflow"

# Set this to 1 to use the nightly (latest) build of TensorFlow instead of a
# named release version. Set it to an already existing value in the environment
# (allowing command line control of the build), defaulting to 0 (release build).
export DV_TF_NIGHTLY_BUILD="${DV_TF_NIGHTLY_BUILD:-0}"

# Set this to 1 to make our prereq scripts install the CUDA libraries.
# If you already have CUDA installed, such as on a properly provisioned
# Docker image, it shouldn't be necessary.
export DV_INSTALL_GPU_DRIVERS="${DV_INSTALL_GPU_DRIVERS:-0}"

export PYTHON_BIN_PATH=$(which python)
export USE_DEFAULT_PYTHON_LIB_PATH=1
export DV_COPT_FLAGS="--copt=-msse4.1 --copt=-msse4.2 --copt=-mavx --copt=-O3"

function note_build_stage {
  echo "========== [$(date)] Stage '${1}' starting"
}

以下是 run-prereq.sh 脚本

#!/bin/bash

# Copyright 2017 Google Inc.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#
# 1. Redistributions of source code must retain the above copyright notice,
#    this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# 3. Neither the name of the copyright holder nor the names of its
#    contributors may be used to endorse or promote products derived from this
#    software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.

# This installs all the libraries (python, dso, etc) that are needed
# by DeepVariant at runtime (except for tensorflow, which is special).
# Some extra stuff may also be included.

set -euo pipefail

echo ========== Load config settings.

source settings.sh

################################################################################
# misc setup
################################################################################

note_build_stage "Misc setup"

if [[ "$EUID" = "0" ]]; then
  # Ensure sudo exists, even if we don't need it.
  apt-get update
  apt-get -y install sudo
fi

note_build_stage "Update package list"

sudo -H apt-get -qq -y update

note_build_stage "Install development packages"

sudo -H apt-get -y install pkg-config zip zlib1g-dev unzip curl

note_build_stage "Install python packaging infrastructure"

sudo -H apt-get -y install python-dev python-pip python-wheel
sudo -H pip install --upgrade pip

################################################################################
# python packages
################################################################################

note_build_stage "Install python packages"

sudo -H pip install contextlib2
sudo -H pip install enum34
sudo -H pip install intervaltree
sudo -H pip install 'mock>=2.0.0'
sudo -H pip install 'numpy==1.12'
sudo -H pip install 'requests>=2.18'
sudo -H pip install 'scipy==1.0'
sudo -H pip install 'oauth2client>=4.0.0'
sudo -H pip install 'crcmod>=1.7'
sudo -H pip install six

################################################################################
# TensorFlow
################################################################################

note_build_stage "Install TensorFlow pip package"

if [[ "${DV_USE_PREINSTALLED_TF}" = "1" ]]; then
  echo "Skipping TensorFlow installation at user request; will use pre-installed TensorFlow."
else
  # Also pip install the latest TensorFlow with cpu support. We don't build the
  # full TF from source, but instead using prebuilt version. However, we still
  # need the full source version to build DeepVariant.

  # Gets the nightly TF build: https://pypi.python.org/pypi/tf-nightly which is
  # necessary right now if we aren't pinning the TF source. We have observed
  # runtime failures if there's too much skew between the released TF package and
  # the source.
  if [[ "${DV_TF_NIGHTLY_BUILD}" = "1" ]]; then
    if [[ "${DV_GPU_BUILD}" = "1" ]]; then
      echo "Installing GPU-enabled TensorFlow nightly wheel"
      sudo -H pip install --upgrade tf_nightly_gpu
    else
      echo "Installing CPU-only TensorFlow nightly wheel"
      sudo -H pip install --upgrade tf_nightly
    fi
  else
    # Use the official TF release pip package.
    if [[ "${DV_GPU_BUILD}" = "1" ]]; then
      echo "Installing GPU-enabled TensorFlow wheel"
      sudo -H pip install --upgrade 'tensorflow-gpu==1.4'
    elif [[ "${DV_USE_GCP_OPTIMIZED_TF_WHL}" = "1" ]]; then
      echo "Installing Google Cloud Platform optimized CPU-only TensorFlow wheel"
      curl "${GCP_OPTIMIZED_TF_WHL_CURL_PATH}/${GCP_OPTIMIZED_TF_WHL_FILENAME}" \
        > "/tmp/${GCP_OPTIMIZED_TF_WHL_FILENAME}"
      sudo -H pip install --upgrade "/tmp/${GCP_OPTIMIZED_TF_WHL_FILENAME}"
    else
      echo "Installing standard CPU-only TensorFlow wheel"
      sudo -H pip install --upgrade 'tensorflow==1.4'
    fi
  fi
fi


################################################################################
# CUDA
################################################################################

if [[ "${DV_GPU_BUILD}" = "1" ]]; then
  if [[ "${DV_INSTALL_GPU_DRIVERS}" = "1" ]]; then
    if [[ "$(lsb_release -d)" != *Ubuntu*16.*.* ]]; then
      echo "CUDA installation only configured for Ubuntu 16"
      exit 1
    fi

    # from https://cloud.google.com/compute/docs/gpus/add-gpus
    echo "Checking for CUDA..."
    if ! dpkg-query -W cuda-8-0; then
      echo "Installing CUDA..."
      CUDA_DEB="cuda-repo-ubuntu1604_8.0.61-1_amd64.deb"
      curl -O http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/${CUDA_DEB}
      sudo -H dpkg -i ./cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
      sudo -H apt-get update
      sudo -H apt-get -y install cuda-8-0
    fi

    echo "Checking for CUDNN..."
    if [[ ! -e /usr/local/cuda-8.0/include/cudnn.h ]]; then
      echo "Installing CUDNN..."
      CUDNN_TAR_FILE="cudnn-8.0-linux-x64-v6.0.tgz"
      wget http://developer.download.nvidia.com/compute/redist/cudnn/v6.0/${CUDNN_TAR_FILE}
      tar -xzvf ${CUDNN_TAR_FILE}
      sudo cp -P cuda/include/cudnn.h /usr/local/cuda-8.0/include
      sudo cp -P cuda/lib64/libcudnn* /usr/local/cuda-8.0/lib64/
      sudo chmod a+r /usr/local/cuda-8.0/lib64/libcudnn*
      sudo ldconfig
    fi

    # Tensorflow says to do this.
    sudo -H apt-get -y install libcupti-dev
  fi

  # If we are doing a gpu-build, nvidia-smi should be install. Run it so we
  # can see what gpu is installed.
  nvidia-smi || :
fi


################################################################################
# Misc dependencies
################################################################################

note_build_stage "Install other packages"

# for htslib
sudo -H apt-get -y install libssl-dev libcurl4-openssl-dev liblz-dev libbz2-dev liblzma-dev

# for the debruijn graph
sudo -H apt-get -y install libboost-graph-dev

note_build_stage "run-prereq.sh complete"

以下是 build_and_test.sh 脚本

#!/bin/bash

# Copyright 2017 Google Inc.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#
# 1. Redistributions of source code must retain the above copyright notice,
#    this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# 3. Neither the name of the copyright holder nor the names of its
#    contributors may be used to endorse or promote products derived from this
#    software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.

# NOLINT
set -eux -o pipefail

source settings.sh

# Run all deepvariant tests.  Take bazel options from args, if any.
# Note: If running with GPU, tests must be executed serially due to a GPU
# contention issue.
if [[ "${DV_GPU_BUILD:-0}" = "1" ]]; then
  bazel test -c opt --local_test_jobs=1 ${DV_COPT_FLAGS} "$@" \
    deepvariant/...
  # Also run the GPU-specific tests.
  bazel test -c opt --local_test_jobs=1 ${DV_COPT_FLAGS} "$@" \
    deepvariant:gpu_tests
else
  # Running parallel tests on CPU.
  bazel test -c opt ${DV_COPT_FLAGS} "$@" deepvariant/...
fi

# Create some build artifacts.
bazel build -c opt ${DV_COPT_FLAGS} "$@" deepvariant:binaries

# Bundle the licenses.
bazel build :licenses_zip

不是 bazel 专家,不过这个错误看着好像是有依赖无法找到的问题。

(14:19:27) ERROR: /home/suanfa/Downloads/deepvariant-r0.6/third_party/nucleus/io/BUILD:400:1: no such package '@com_google_absl//absl/memory': The repository could not be resolved and referenced by '//third_party/nucleus/io:vcf_writer'
(14:19:27) ERROR: Analysis of target '//deepvariant/labeler:labeled_examples_to_vcf' failed; build aborted: no such package '@com_google_absl//absl/memory': The repository could not be resolved
(14:19:27) INFO: Elapsed time: 33.556s (14:19:27) FAILED: Build did NOT complete successfully (53 packages loaded)
(14:19:27) ERROR: Couldn't start the build. Unable to run tests

@com_google_absl//absl/memory 这个包要能解析,首先 bazel 要知道 com_google_absl 能够解析到哪一个 GitHub repo,这个 mapping 一般是在 workspace 里面给出的,也就是 deepvariant/WORKSPACE at r0.6 · google/deepvariant · GitHub 这个文件

# Abseil libraries
git_repository (
    name = "com_google_absl_py",
    # redacted
    commit = "5e343642d987268df199b4c851b7dd3d687ac316",
    remote = "https://github.com/abseil/abseil-py.git",
)
# Note: com_google_absl (the C++ abseil library) is provided by TensorFlow.

可以看到,其中定义了 com_google_absl_py 这个名称,但是 com_google_absl 是从 tensorflow 载入的。

我们继续往下看,可以看到 bazel 将会载入 tensorflow 目录的 tensorflow/workspace.bzl 并将其用作 org_tensorflow

# Import all of the tensorflow dependencies.
load ("@org_tensorflow//tensorflow:workspace.bzl", "tf_workspace")

# Import tensorflow.  Note path.
local_repository (
    name = "org_tensorflow",
    path = "../tensorflow",
)

那么我们看一下 tensorflow/workspace.bzl 中是否对 com_google_absl 有定义。

果然 https://github.com/tensorflow/tensorflow/blob/master/tensorflow/workspace.bzl 定义了这个名称

  tf_http_archive (
      name = "com_google_absl",
      urls = [
          "https://mirror.bazel.build/github.com/abseil/abseil-cpp/archive/720c017e30339fd1786ce4aac68bc8559736e53f.tar.gz",
          "https://github.com/abseil/abseil-cpp/archive/720c017e30339fd1786ce4aac68bc8559736e53f.tar.gz",
      ],
     sha256 = "5996380e3e8b981f55d1c8d58e709c00dbb4806ba367be75d0925a68cc2f6478",
     strip_prefix = "abseil-cpp-720c017e30339fd1786ce4aac68bc8559736e53f",
     build_file = clean_dep ("//third_party:com_google_absl.BUILD"),
  )

理清了思路回头看一下这个问题。我猜测很有肯能是 bazel 没有在 tensorflow/workspace.bzl 里面看到 com_google_absl

建议可以看一下父目录是否有 tensorflow 的源代码。

如果没有的话也可以通过这个脚本生成 https://github.com/google/deepvariant/blob/r0.6/build-prereq.sh

(cd .. &&
 git clone [url=https://github.com/tensorflow/tensorflow] https://github.com/tensorflow/tensorflow [/url] &&
 cd tensorflow &&
 git checkout v1.7.0 &&
 echo | ./configure)

另外关于 1) 我觉得可能是 bazel 12 和 deepvariant 不兼容。Deepvariant 自带的 bazel 也是 11 版本。


舟 3332,发表于 2018-4-25 17:40:51

我已经尝试卸载 bazel 12,然后安装 bazel 11,也是不行。另外关于问题 2),我同事是已经运行脚本上 build-prereq.sh,已经在 DeepVariant-r0.6 的父目录建立 tensorflow 的源代码。我出现的问题就是找不到父目录上的 Tensorflow 目录中的"/home/lujianliang/deepvariant-r0.6/…/tensorflow/.tf_configure.bazelrc"。


Simon0808(提问者),2018-4-26 20:37

你用的是哪个版本? deepvariant master branch 上的最新版吗?


舟 3332,2018-4-28 07:13

我使用的是 DeepVariant 的 0.6 版本。我发现一个问题:DeepVariant0.6 版本的源代码中的 run-prereq.sh 是安装 TensorflowGPU1.4 版本的,cuda8.0,cudnn v6。我的服务器上也是安装这些版本。但是 build-prereq.sh 这个脚本的最后一段代码是先下载 Tensorflow 的源代码,然后调用"git checkout v1.7.0"来构建最新的 Tensorflow1.7 版本。那么如果构建 Tensorflow GPU1.7 版本,Tensorflow 的官网介绍是需要 cuda9.0 以及 cudnn v7 的吧。代码是 “(cd … &&git clone GitHub - tensorflow/tensorflow: An Open Source Machine Learning Framework for Everyone &&cd tensorflow && git checkout v1.7.0 && echo | ./configure)”。另外一个与我一起尝试安装 DeepVariant 的中科大的人员使用的是 tensorflow 1.6 版本,他发现 Tensorflow1.4 版本和 1.7 版本的 workspace 有很大差别,1.6 版本跟 1.7 版本较为接近,他使用 Tensorflow 1.6 版本以及通过 build_and_test.sh 的 33 个 test 中 10 个 test。因为他的机器也是安装 cuda8.0 以及 cudnn v6,Tensorflow 1.7 版本也不能使用。


Simon0808(提问者),2018-4-29 08:43

Hi, thanks for your question.
你觀察到的和前面的回答都提到重點: DeepVariant 0.6 currently works with Bazel version 0.11.0, but not 0.12.0.
build-prereq.sh 裡面設定 bazel 版本為 0.11.0, 所以理論上如果你先跑過 build-prereq.sh 這個 script, 他應該會安裝 0.11.0

你有試過先執行:
build-prereq.sh
然後再用 build_and_test.sh 嗎?

在 bazel 版本為 0.11.0 的狀況之下,是否可以 build 呢?

我目前測試的環境是使用 Ubuntu 16.04, 基本上直接使用 Case Study 建議的設定從 GCP 上取得一個 Ubuntu 16.04 的機器:

在這樣的環境下,如果我 clone DeepVariant 的 codebase, 然後跑

./build-prereq.sh
./build_and_test.sh

的話,是可以 build 的。不過我也測試了:如果我將 bazel 升級到 0.12.0, 就會產生錯誤。

我會再研究看看如何在未來版本裡 upgrade bazel version. Thanks for your feeback!!!


pichuan,发表于 2018-4-26 03:01:30

感谢您给出的帮助。我已经尝试卸载 bazel 12,重新安装 bazel 11。但是还是出现了同样的问题:找不到父目录上的 Tensorflow 目录中的"/home/lujianliang/deepvariant-r0.6/…/tensorflow/.tf_configure.bazelrc"。同时注意到你的一句话 “基本上直接使用 Case Study 建議的設定從 GCP 上取得一個 Ubuntu 16.04 的機器”,我理解您的意思是从 GCP 下载取得那些已经编译好 zip 文件运行 DeepVariant 的三个步骤吗?其实我们是已经可以从 GCP 下载 make_examples.zip,call_variants.zip 和 postprocess_variants.zip 这个三个 zip 文件对 BAM 文件进行变异检测了。我们的问题是如果我们想修改 DeepVariant 的代码,然后是不是需要通过 bazel 对 py 脚本进行编译才能生成类似的 zip 文件然后对 bam 文件进行变异检测。我们已经试过直接使用 “python make_examples.py ” 这样方式运行,但是会出现一些缺少依赖库的报错,所以我们在想是跟 bazel 没有完全安装或者 bazel 的使用方法不正确所造成的。希望把 bazel 给安装准确。


Simon0808(提问者),2018-4-26 21:31

pichuan 应该说的是直接从 GCP 上开 Ubuntu 的虚拟机跑没有问题。


舟 3332,2018-4-28 07:14

Hi,

I just released a new update:

which hopefully fixes your build problem on Debian. (I tested it on a Debian machine and it builds for me now)

Can you give it a try and see if you can build it on Debian now?


By pichuan, 2018-5-1 12:12