Skip to content

Commit

Permalink
add Marvin FCN, update Realsense ROS, update README
Browse files Browse the repository at this point in the history
  • Loading branch information
andyzeng committed Aug 19, 2016
1 parent 49b9215 commit af39d2b
Show file tree
Hide file tree
Showing 14 changed files with 8,693 additions and 23 deletions.
65 changes: 60 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ UNFINISHED ... PLEASE DO NOT USE
## Documentation
* [Realsense Standalone](#realsense-standalone)
* [Realsense ROS Package](#realsense-ros-package)
* [Marvin FCN ROS Package](#marvin-fcn-ros-package)

## Realsense Standalone

Expand Down Expand Up @@ -60,14 +61,68 @@ See `ros_packages/realsense_camera`
* Used for saving point clouds

### Compilation
* Copy the ROS package `ros_packages/realsense_camera` into your catkin workspace source directory (e.g. `catkin_ws/src`)
* If necessary, configure `realsense_camera/CMakeLists.txt` according to your respective dependencies
* In your catkin workspace, compile the package with `catkin_make`
* Source `devel/setup.sh`
1. Copy the ROS package `ros_packages/realsense_camera` into your catkin workspace source directory (e.g. `catkin_ws/src`)
2. If necessary, configure `realsense_camera/CMakeLists.txt` according to your respective dependencies
3. In your catkin workspace, compile the package with `catkin_make`
4. Source `devel/setup.sh`

### Usage
* Start `roscore`
* Run `rosrun realsense_camera capture` to stream data from the sensor and start the data capture service:
* To start the RGB-D data capture service and stream data from the sensor, run:

```shell
rosrun realsense_camera capture
```

* The service `/realsense_camera` returns data from the sensor (response data format described in `realsense_camera/srv/StreamSensor.srv`)
* If you need a GL window to see the streamed RGB-D data, run `rosrun realsense_camera capture _display:=True`

## Marvin FCN ROS Package

A C++ ROS package for deep learning based object segmentation using [FCNs (Fully Convolutional Networks)](https://arxiv.org/abs/1411.4038) with [Marvin](http://marvin.is/), a lightweight GPU-only neural network framework. This package feeds RGB-D data forward through a pre-trained ConvNet to retrieve object segmentation results. The neural networks are trained offline with [Marvin](http://marvin.is/).

See `ros_packages/marvin_fcn`

### Dependencies

1. [Realsense ROS Package](#realsense-ros-package)

2. [CUDA 7.5](https://developer.nvidia.com/cuda-downloads) and [cuDNN 5](https://developer.nvidia.com/cudnn) to support [Marvin](http://marvin.is/). You may need to register with NVIDIA. Below are some additional steps to set up cuDNN 5. **NOTE** We highly recommend that you install different versions of cuDNN to different directories (e.g., ```/usr/local/cudnn/vXX```) because different software packages may require different versions.

```shell
LIB_DIR=lib$([[ $(uname) == "Linux" ]] && echo 64)
CUDNN_LIB_DIR=/usr/local/cudnn/v5/$LIB_DIR
echo LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CUDNN_LIB_DIR >> ~/.profile && ~/.profile

tar zxvf cudnn*.tgz
sudo cp cuda/$LIB_DIR/* $CUDNN_LIB_DIR/
sudo cp cuda/include/* /usr/local/cudnn/v5/include/
```
3. OpenCV (tested with OpenCV 2.4.11)
* Used for saving images

### Compilation
1. Copy the ROS package `ros_packages/marvin_fcn` into your catkin workspace source directory (e.g. `catkin_ws/src`)
2. If necessary, configure `realsense_camera/CMakeLists.txt` according to your respective dependencies
3. In your catkin workspace, compile the package with `catkin_make`
4. Source `devel/setup.sh`

change location of where net is loaded

make sure where data is going to be read and written

ros package to compute hha

`rosrun marvin_fcn detect _service_mode:=1 _write_directory:="/home/andyz/apc/toolbox/ros_pacges/catkin_ws/src/marvin_fcn/data"`

`rosrun marvin_fcn detect _service_mode:=2 _read_directory:="/home/andyz/apc/toolbox/ros_packages/catkin_ws/src/marvin_fcn/data" _write_directory:="/home/andyz/apc/toolbox/ros_packages/catkin_ws/src/marvin_fcn/data"`

`rosservice call /marvin_fcn ["elmers_washable_no_run_school_glue","expo_dry_erase_board_eraser"] 0 0`








199 changes: 199 additions & 0 deletions ros_packages/marvin_fcn/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,199 @@
cmake_minimum_required(VERSION 3.1.0)
set(CMAKE_CXX_STANDARD 11)
project(marvin_fcn)

## Find catkin macros and libraries
## IF COMPONENTS list like find_package(catkin REQUIRED COMPONENTS xyz)
## is used, also find other catkin packages

## System dependencies are found with CMake's conventions
find_package(catkin REQUIRED COMPONENTS
roscpp
message_generation
geometry_msgs
std_msgs
)

find_package(OpenCV REQUIRED)

# CUDA and CUDNN
find_package(CUDA REQUIRED)
set(CUDA_HOST_COMPILER "c++")
set(CUDA_NVCC_FLAGS "-std=c++11")
set(CUDNN_INCLUDE_DIR /usr/local/cudnn/v5/include CACHE PATH "Path to cudnn header file")
set(CUDNN_LIB_DIR /usr/local/cudnn/v5/lib64)

## Uncomment this IF the package has a setup.py. This macro ensures
## modules and global scripts declared therein get installed
## See http://ros.org/doc/api/catkin/html/user_guide/setup_dot_py.html
# catkin_python_setup()

################################################
## Declare ROS messages, services and actions ##
################################################

## To declare and build messages, services or actions from within this
## package, follow these steps:
## * Let MSG_DEP_SET be the set of packages whose message types you use in
## your messages/services/actions (e.g. std_msgs, actionlib_msgs, ...).
## * In the file package.xml:
## * add a build_depend tag for "message_generation"
## * add a build_depend and a run_depend tag for each package in MSG_DEP_SET
## * If MSG_DEP_SET isn't empty the following dependency has been pulled in
## but can be declared for certainty nonetheless:
## * add a run_depend tag for "message_runtime"
## * In this file (CMakeLists.txt):
## * add "message_generation" and every package in MSG_DEP_SET to
## find_package(catkin REQUIRED COMPONENTS ...)
## * add "message_runtime" and every package in MSG_DEP_SET to
## catkin_package(CATKIN_DEPENDS ...)
## * uncomment the add_*_files sections below as needed
## and list every .msg/.srv/.action file to be processed
## * uncomment the generate_messages entry below
## * add every package in MSG_DEP_SET to generate_messages(DEPENDENCIES ...)

## Generate messages in the 'msg' folder
add_message_files(
FILES
# ObjectHypothesis.msg
)

## Generate services in the 'srv' folder
add_service_files(
FILES
DetectObjects.srv
)

## Generate actions in the 'action' folder
# add_action_files(
# FILES
# Action1.action
# Action2.action
# )

## Generate added messages and services with any dependencies listed here
generate_messages(
DEPENDENCIES
std_msgs # Or other packages containing msgs
geometry_msgs
)

################################################
## Declare ROS dynamic reconfigure parameters ##
################################################

## To declare and build dynamic reconfigure parameters within this
## package, follow these steps:
## * In the file package.xml:
## * add a build_depend and a run_depend tag for "dynamic_reconfigure"
## * In this file (CMakeLists.txt):
## * add "dynamic_reconfigure" to
## find_package(catkin REQUIRED COMPONENTS ...)
## * uncomment the "generate_dynamic_reconfigure_options" section below
## and list every .cfg file to be processed

## Generate dynamic reconfigure parameters in the 'cfg' folder
# generate_dynamic_reconfigure_options(
# cfg/DynReconf1.cfg
# cfg/DynReconf2.cfg
# )

###################################
## catkin specific configuration ##
###################################
## The catkin_package macro generates cmake config files for your package
## Declare things to be passed to dependent projects
## INCLUDE_DIRS: uncomment this IF you package contains header files
## LIBRARIES: libraries you create in this project that dependent projects also need
## CATKIN_DEPENDS: catkin_packages dependent projects also need
## DEPENDS: system dependencies of this project that dependent projects also need
# catkin_package(
# INCLUDE_DIRS include
# LIBRARIES apc_vision
# CATKIN_DEPENDS other_catkin_pkg
# DEPENDS system_lib
# )

###########
## Build ##
###########

## Specify additional locations of header files
## Your package locations should be listed before other locations
# include_directories(include)

## Declare a C++ library
# add_library(apc_vision
# src/${PROJECT_NAME}/apc_vision.cpp
# )

## Add cmake target dependencies of the library
## as an example, code may need to be generated before libraries
## either from message generation or dynamic reconfigure
# add_dependencies(apc_vision ${${PROJECT_NAME}_EXPORTED_TARGETS} ${catkin_EXPORTED_TARGETS})

# Add C++ header/source files
SET(HEADERS
include/depth_utils.h
)
SET(SOURCES
src/detect.cu
src/marvin.hpp
)

## Declare a C++ executable
include_directories(include /usr/local/include ${catkin_INCLUDE_DIRS} ${OpenCV_INCLUDE_DIR} ${CUDNN_INCLUDE_DIR})
link_directories(${CUDNN_LIB_DIR} ${OpenCV_LIB_DIR})
cuda_add_executable(detect ${SOURCES} ${HEADERS})
target_link_libraries(detect ${catkin_LIBRARIES} realsense ${OpenCV_LIBS}
pthread cudnn ${CUDA_CUBLAS_LIBRARIES} ${CUDA_LIBRARIES} ${CUDA_curand_LIBRARY})
add_dependencies(detect ${PROJECT_NAME}_gencpp)
add_dependencies(detect ${PROJECT_NAME}_generate_messages_cpp)

#############
## Install ##
#############

# all install targets should use catkin DESTINATION variables
# See http://ros.org/doc/api/catkin/html/adv_user_guide/variables.html

## Mark executable scripts (Python etc.) for installation
## in contrast to setup.py, you can choose the destination
# install(PROGRAMS
# scripts/my_python_script
# DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
# )

## Mark executables and/or libraries for installation
# install(TARGETS apc_vision apc_vision
# ARCHIVE DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION}
# LIBRARY DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION}
# RUNTIME DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
# )

## Mark cpp header files for installation
# install(DIRECTORY include/${PROJECT_NAME}/
# DESTINATION ${CATKIN_PACKAGE_INCLUDE_DESTINATION}
# FILES_MATCHING PATTERN "*.h"
# PATTERN ".svn" EXCLUDE
# )

## Mark other files for installation (e.g. launch and bag files, etc.)
# install(FILES
# # myfile1
# # myfile2
# DESTINATION ${CATKIN_PACKAGE_SHARE_DESTINATION}
# )

#############
## Testing ##
#############

## Add gtest based cpp test target and link libraries
# catkin_add_gtest(${PROJECT_NAME}-test test/test_apc_vision.cpp)
# IF(TARGET ${PROJECT_NAME}-test)
# target_link_libraries(${PROJECT_NAME}-test ${PROJECT_NAME})
# ENDIF()

## Add folders to be run by python nosetests
# catkin_add_nosetests(test)
25 changes: 25 additions & 0 deletions ros_packages/marvin_fcn/include/depth_utils.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#include <opencv2/opencv.hpp>

// Depth images are read and written as 16-bit PNG files, with
// depth in decimillimeters (10^-4 meters) and circularly
// bitshifted to the right by 3 bits.
//
// WriteDepth takes a row-major order float array of depth
// values and saves it as a depth image using OpenCV.

// ReadDepth reads a depth image and returns a row-major order
// float array of depth values.

void WriteDepth(const std::string &depth_file, float * depth_values, int frame_height, int frame_width) {
cv::Mat depth_mat(frame_height, frame_width, CV_16UC1);
for (size_t y = 0; y < frame_height; y++)
for (size_t x = 0; x < frame_width; x++) {
unsigned short depth_short = (unsigned short)(depth_values[y * frame_width + x] * 10000);
depth_short = (depth_short >> 13 | depth_short << 3);
depth_mat.at<unsigned short>(y, x) = depth_short;
}
std::vector<int> compression_params;
compression_params.push_back(CV_IMWRITE_PNG_COMPRESSION);
compression_params.push_back(9);
cv::imwrite(depth_file, depth_mat, compression_params);
}
54 changes: 54 additions & 0 deletions ros_packages/marvin_fcn/package.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
<?xml version="1.0.0"?>
<package>
<name>marvin_fcn</name>
<version>1.0.0</version>
<description>The marvin_fcn package</description>

<!-- One maintainer tag required, multiple allowed, one person per tag -->
<!-- Example: -->
<!-- <maintainer email="[email protected]">Jane Doe</maintainer> -->
<maintainer email="[email protected]">Andy Zeng</maintainer>
<maintainer email="[email protected]">Jianxiong Xiao</maintainer>
<maintainer email="[email protected]">Peter K.T. Yu</maintainer>


<!-- One license tag required, multiple allowed, one license per tag -->
<!-- Commonly used license strings: -->
<!-- BSD, MIT, Boost Software License, GPLv2, GPLv3, LGPLv2.1, LGPLv3 -->
<license>MIT</license>


<!-- Url tags are optional, but mutiple are allowed, one per tag -->
<!-- Optional attribute type can be: website, bugtracker, or repository -->
<!-- Example: -->
<!-- <url type="website">http://wiki.ros.org/apc_vision</url> -->


<!-- Author tags are optional, mutiple are allowed, one per tag -->
<!-- Authors do not have to be maintianers, but could be -->
<!-- Example: -->
<!-- <author email="[email protected]">Jane Doe</author> -->


<!-- The *_depend tags are used to specify dependencies -->
<!-- Dependencies can be catkin packages or system dependencies -->
<!-- Examples: -->
<!-- Use build_depend for packages you need at compile time: -->
<build_depend>message_generation</build_depend>
<build_depend>std_msgs</build_depend>
<!-- Use buildtool_depend for build tool packages: -->
<!-- <buildtool_depend>catkin</buildtool_depend> -->
<!-- Use run_depend for packages you need at runtime: -->
<run_depend>message_runtime</run_depend>
<!-- Use test_depend for packages you need only for testing: -->
<!-- <test_depend>gtest</test_depend> -->
<buildtool_depend>catkin</buildtool_depend>
<build_depend>roscpp</build_depend>
<run_depend>roscpp</run_depend>

<!-- The export tag contains other, unspecified, tags -->
<export>
<!-- Other tools can request additional information be placed here -->

</export>
</package>
Loading

0 comments on commit af39d2b

Please sign in to comment.