Merge branch 'master' into export-saved-model

Author: adamcrume

.travis.yml

@@ -1,23 +1,25 @@
 language: rust
 sudo: false
+dist: trusty # still in beta, but required for the prebuilt TF binaries
 
 cache:
   cargo: true
   directories:
     - $HOME/.cache/bazel
 
-rust: nightly
+rust: stable
 
 install:
   - export CC="gcc-4.9" CXX="g++-4.9"
   - source travis-ci/install.sh
 
 script:
+  - export RUST_BACKTRACE=1
   - cargo test -vv -j 2 --features tensorflow_unstable
   - cargo run --example regression
   - cargo run --features tensorflow_unstable --example expressions
   - cargo doc -vv --features tensorflow_unstable
-  - (cd tensorflow-sys && cargo test -vv -j 1)
+  - # TODO(#66): Re-enable: (cd tensorflow-sys && cargo test -vv -j 1)
   - (cd tensorflow-sys && cargo doc -vv)
 
 addons:

README.md

@@ -1,90 +1,105 @@
-#TensorFlow Rust
+# <img alt="TensorFlow" src="https://www.tensorflow.org/images/tf_logo_transp.png" width="170"/> Rust Binding
 [![Version](https://img.shields.io/crates/v/tensorflow.svg)](https://crates.io/crates/tensorflow)
 [![Status](https://travis-ci.org/tensorflow/rust.svg?branch=master)](https://travis-ci.org/tensorflow/rust)
 
 TensorFlow Rust provides idiomatic [Rust](https://www.rust-lang.org) language
 bindings for [TensorFlow](http://tensorflow.org).
 
-This project is still under active development and not guaranteed to have a
-stable API. This is especially true because the TensorFlow C API used by this
-project has not yet stabilized.
+**Notice:** This project is still under active development and not guaranteed to have a
+stable API. This is especially true because the underlying TensorFlow C API has not yet
+been stabilized as well.
 
-## Using
+* [Documentation](https://tensorflow.github.io/rust/tensorflow/)
+* [TensorFlow website](http://tensorflow.org)
+* [TensorFlow GitHub page](https://github.com/tensorflow/tensorflow)
+
+## Getting Started
+Since this crate depends on the TensorFlow C API, it needs to be compiled first. This crate will
+automatically compile TensorFlow for you, but it is also possible to manually install TensorFlow
+and the crate will pick it up accordingly.
+
+### Prerequisites
+The following dependencies are needed to compile and build this crate (assuming TensorFlow itself
+  should also be compiled transparently):
+
+ - git
+ - [bazel](https://bazel.build/)
+ - Python Dependencies `numpy`, `dev`, `pip` and `wheel`
+ - Optionally, CUDA packages to support GPU-based processing
 
-If you only wish to use TensorFlow within your own project, simply include 
-`tensorflow = "*"` in your `Cargo.toml` file. If the TensorFlow library is 
-not already installed on your computer in a place it can be found, you will
-need the requirements listed in "Automatically Building Tensorflow" below,
-and Cargo will build the library for you.
+The TensorFlow website provides detailed instructions on how to obtain and install said dependencies,
+so if you are unsure please [check out the docs](https://www.tensorflow.org/install/install_sources)
+ for further details.
 
-## Building
+### Usage
+Add this to your `Cargo.toml`:
 
-If you only intend to use TensorFlow from within Rust, then you don't need to
-build TensorFlow manually and can follow the automatic steps. If you do need to
-use TensorFlow outside of Rust, the manual steps will provide you with a
-TensorFlow header file and shared library that can be used by other languages.
+```toml
+[dependencies]
+tensorflow = "0.3"
+```
 
-### Automatically building TensorFlow
+and this to your crate root:
+
+```rust
+extern crate tensorflow;
+```
 
-Install [SWIG](http://www.swig.org) and [NumPy](http://www.numpy.org).  The
-version from your distro's package manager should be fine for these two.  Also
-install [Bazel](http://bazel.io/docs/install.html), which you may need to do
-from source.
 Then run `cargo build -j 1`. Since TensorFlow is built during this process, and
 the TensorFlow build is very memory intensive, we recommend using the `-j 1`
 flag which tells cargo to use only one task, which in turn tells TensorFlow to
 build with only on task. Of course, if you have a lot of RAM, you can use a
 higher value.
+
 To include the especially unstable API (which is currently the `expr` module),
 use `--features tensorflow_unstable`.
 
-### Manually building TensorFlow
+For now, please see the [Examples](https://github.com/tensorflow/rust/tree/master/examples) for more
+details on how to use this binding.
 
-Install [TensorFlow from source](https://www.tensorflow.org/versions/master/get_started/os_setup.html#source).
+## Manual TensorFlow Compilation
+If you don't want to build TensorFlow after every `cargo clean` or you want to work against
+unreleased/unsupported TensorFlow versions, manual compilation is the way to go.
+
+See [TensorFlow from source](https://www.tensorflow.org/install/install_sources) first.
 The Python/pip steps are not necessary, but building `tensorflow:libtensorflow.so` is.
 
 In short:
 
 1. Install [SWIG](http://www.swig.org) and [NumPy](http://www.numpy.org).  The
    version from your distro's package manager should be fine for these two.
-1. [Install Bazel](http://bazel.io/docs/install.html), which you may need to do
+2. [Install Bazel](http://bazel.io/docs/install.html), which you may need to do
    from source.
-1. `git clone --recurse-submodules https://github.com/tensorflow/tensorflow`
-1. `cd tensorflow`
-1. `./configure`
-1. `bazel build -c opt --jobs=1 tensorflow:libtensorflow.so`
+3. `git clone https://github.com/tensorflow/tensorflow`
+4. `cd tensorflow`
+5. `./configure`
+6. `bazel build --compilation_mode=opt --copt=-march=native --jobs=1 tensorflow:libtensorflow.so`
 
-   Using --jobs=1 is recommended unless you have a lot of RAM, because
+   Using `--jobs=1` is recommended unless you have a lot of RAM, because
    TensorFlow's build is very memory intensive.
 
-Copy $TENSORFLOW_SRC/bazel-bin/tensorflow/libtensorflow.so to /usr/local/lib.
-If this is not possible, add $TENSORFLOW_SRC/bazel-bin/tensorflow to
-LD_LIBRARY_PATH.
+Copy `$TENSORFLOW_SRC/bazel-bin/tensorflow/libtensorflow.so` to `/usr/local/lib`.
+If this is not possible, add `$TENSORFLOW_SRC/bazel-bin/tensorflow` to
+`LD_LIBRARY_PATH`.
 
-You may need to run `ldconfig` to reset `ld`'s cache after copying libtensorflow.so.
+You may need to run `ldconfig` to reset `ld`'s cache after copying `libtensorflow.so`.
 
-Now run `cargo build` as usual.
-To include the especially unstable API (which is currently the `expr` module),
-use `--features tensorflow_unstable`.
-
-## RFCs
-RFCs are [issues tagged with RFC](https://github.com/tensorflow/rust/labels/rfc).
-Check them out and comment. Discussions are welcome. After all, thats what a Request For Comment is for!
+**OSX Note**: If you are running on OSX, there is a
+[Homebrew PR](https://github.com/Homebrew/homebrew-core/pull/10273) in process which, once merged,
+will make it easy to install `libtensorflow` wihout hassle. In the meantime, you can take a look at
+[snipsco/tensorflow-build](https://github.com/snipsco/tensorflow-build) which provides a homebrew
+tap that does essentially the same.
 
-## FAQs
+## FAQ's
 
-#### Why does the compiler say that parts of the API don't exist?
+### Why does the compiler say that parts of the API don't exist?
 The especially unstable parts of the API (which is currently the `expr` modul) are
 feature-gated behind the feature `tensorflow_unstable` to prevent accidental
 use. See http://doc.crates.io/manifest.html#the-features-section.
 (We would prefer using an `#[unstable]` attribute, but that
 [doesn't exist](https://github.com/rust-lang/rfcs/issues/1491) yet.)
 
-## Other
-
-This project is not directly affiliated with the TensorFlow project, although we
-do intend to communicate and cooperate with them.
-
+## Contributing
 Developers and users are welcome to join
 [#tensorflow-rust](http://chat.mibbit.com/?server=irc.mozilla.org&channel=%23tensorflow-rust)
 on irc.mozilla.org.
@@ -93,8 +108,9 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for information on how to contribute code
 
 This is not an official Google product.
 
-##For more information
+RFCs are [issues tagged with RFC](https://github.com/tensorflow/rust/labels/rfc).
+Check them out and comment. Discussions are welcome. After all, thats what a Request For
+Comment is for!
 
-* [API docs](https://tensorflow.github.io/rust)
-* [TensorFlow website](http://tensorflow.org)
-* [TensorFlow GitHub page](https://github.com/tensorflow/tensorflow)
+## License
+This project is licensed under the terms of the [Apache 2.0 license](https://github.com/tensorflow/rust/blob/master/LICENSE).

src/buffer.rs

@@ -17,6 +17,7 @@ use std::ops::Range;
 use std::ops::RangeFrom;
 use std::ops::RangeFull;
 use std::ops::RangeTo;
+use std::os::raw::c_void as std_c_void;
 use std::ptr;
 use std::slice;
 use super::BufferTrait;
@@ -65,7 +66,7 @@ impl<T: TensorType> Buffer<T> {
             let msg = CStr::from_ptr(c_msg);
             panic!("Failed to allocate: {}", msg.to_str().unwrap());
         }
-        (*inner).data = ptr;
+        (*inner).data = ptr as *mut std_c_void;
         (*inner).length = len;
         (*inner).data_deallocator = Some(deallocator);
         Buffer {
@@ -81,7 +82,7 @@ impl<T: TensorType> Buffer<T> {
     /// The underlying data is *not* freed when the buffer is destroyed.
     pub unsafe fn from_ptr(ptr: *mut T, len: usize) -> Self {
         let inner = tf::TF_NewBuffer();
-        (*inner).data = ptr as *const c_void;
+        (*inner).data = ptr as *const std_c_void;
         (*inner).length = len;
         Buffer {
             inner: inner,
@@ -155,8 +156,8 @@ impl<T: TensorType> BufferTrait for Buffer<T> {
     }
 }
 
-unsafe extern "C" fn deallocator(data: *mut c_void, _length: size_t) {
-    libc::free(data);
+unsafe extern "C" fn deallocator(data: *mut std_c_void, _length: size_t) {
+    libc::free(data as *mut c_void);
 }
 
 impl<T: TensorType> Drop for Buffer<T> {

src/graph.rs

@@ -9,6 +9,7 @@ use std;
 use std::ffi::CStr;
 use std::ffi::CString;
 use std::ffi::NulError;
+use std::os::raw::c_void as std_c_void;
 use std::ptr;
 use std::str::Utf8Error;
 use std::sync::Arc;
@@ -302,7 +303,7 @@ impl Operation {
     pub fn output_type(&self, index: usize) -> DataType {
         unsafe {
             DataType::from_c(tf::TF_OperationOutputType(tf::TF_Output {
-                operation: self.inner,
+                oper: self.inner,
                 index: index as c_int,
             }))
         }
@@ -332,7 +333,7 @@ impl Operation {
     pub fn input_type(&self, index: usize) -> DataType {
         unsafe {
             DataType::from_c(tf::TF_OperationInputType(tf::TF_Input {
-                operation: self.inner,
+                oper: self.inner,
                 index: index as c_int,
             }))
         }
@@ -359,11 +360,11 @@ impl Operation {
     pub fn input(&self, index: usize) -> (Operation, usize) {
         unsafe {
             let port = tf::TF_OperationInput(tf::TF_Input {
-                operation: self.inner,
+                oper: self.inner,
                 index: index as c_int,
             });
             (Operation {
-                 inner: port.operation,
+                 inner: port.oper,
                  gimpl: self.gimpl.clone(),
              },
              port.index as usize)
@@ -374,7 +375,7 @@ impl Operation {
     pub fn output_num_consumers(&self, index: usize) -> usize {
         unsafe {
             tf::TF_OperationOutputNumConsumers(tf::TF_Output {
-                operation: self.inner,
+                oper: self.inner,
                 index: index as c_int,
             }) as usize
         }
@@ -387,12 +388,12 @@ impl Operation {
     pub fn output_consumers(&self, index: usize) -> Vec<(Operation, usize)> {
         unsafe {
             let num_consumers = tf::TF_OperationOutputNumConsumers(tf::TF_Output {
-                operation: self.inner,
+                oper: self.inner,
                 index: index as c_int,
             });
             let mut vec = <Vec<tf::TF_Input>>::with_capacity(num_consumers as usize);
             let len = tf::TF_OperationOutputConsumers(tf::TF_Output {
-                                                          operation: self.inner,
+                                                          oper: self.inner,
                                                           index: index as c_int,
                                                       },
                                                       vec.as_mut_ptr(),
@@ -401,7 +402,7 @@ impl Operation {
             vec.into_iter()
                 .map(|port| {
                     (Operation {
-                         inner: port.operation,
+                         inner: port.oper,
                          gimpl: self.gimpl.clone(),
                      },
                      port.index as usize)
@@ -483,7 +484,7 @@ pub struct Input<'a> {
 impl<'a> Input<'a> {
     fn to_c(&self) -> tf::TF_Input {
         tf::TF_Input {
-            operation: self.operation.inner,
+            oper: self.operation.inner,
             index: self.index,
         }
     }
@@ -505,7 +506,7 @@ pub struct Output<'a> {
 impl<'a> Output<'a> {
     fn to_c(&self) -> tf::TF_Output {
         tf::TF_Output {
-            operation: self.operation.inner,
+            oper: self.operation.inner,
             index: self.index,
         }
     }
@@ -606,7 +607,7 @@ impl<'a> OperationDescription<'a> {
         unsafe {
             tf::TF_SetAttrString(self.inner,
                                  c_attr_name.as_ptr(),
-                                 c_value.as_ptr() as *const c_void,
+                                 c_value.as_ptr() as *const std_c_void,
                                  c_value.len() as size_t);
         }
         Ok(())
@@ -625,7 +626,7 @@ impl<'a> OperationDescription<'a> {
         unsafe {
             tf::TF_SetAttrStringList(self.inner,
                                      c_attr_name.as_ptr(),
-                                     ptrs.as_ptr(),
+                                     ptrs.as_ptr() as *const *const std_c_void,
                                      lens.as_ptr(),
                                      ptrs.len() as c_int);
         }
@@ -821,7 +822,7 @@ impl<'a> OperationDescription<'a> {
         unsafe {
             tf::TF_SetAttrTensorShapeProto(self.inner,
                                            c_attr_name.as_ptr(),
-                                           value.as_ptr() as *const c_void,
+                                           value.as_ptr() as *const std_c_void,
                                            value.len() as size_t,
                                            status.inner());
         }
@@ -843,7 +844,7 @@ impl<'a> OperationDescription<'a> {
         unsafe {
             tf::TF_SetAttrTensorShapeProtoList(self.inner,
                                                c_attr_name.as_ptr(),
-                                               ptrs.as_ptr(),
+                                               ptrs.as_ptr() as *const *const std_c_void,
                                                lens.as_ptr(),
                                                ptrs.len() as c_int,
                                                status.inner());
@@ -878,7 +879,7 @@ impl<'a> OperationDescription<'a> {
             let ptrs: Vec<*mut tf::TF_Tensor> = value.into_iter().map(|x| x.into_ptr()).collect();
             tf::TF_SetAttrTensorList(self.inner,
                                      c_attr_name.as_ptr(),
-                                     ptrs.as_ptr(),
+                                     ptrs.as_ptr() as *const *const tf::TF_Tensor,
                                      ptrs.len() as c_int,
                                      status.inner());
         }
@@ -893,7 +894,7 @@ impl<'a> OperationDescription<'a> {
         unsafe {
             tf::TF_SetAttrValueProto(self.inner,
                                      c_attr_name.as_ptr(),
-                                     value.as_ptr() as *const c_void,
+                                     value.as_ptr() as *const std_c_void,
                                      // Allow trivial_numeric_casts because usize is not
                                      // necessarily size_t.
                                      value.len() as size_t,

src/lib.rs

@@ -30,6 +30,7 @@ use std::ops::Deref;
 use std::ops::DerefMut;
 use std::ops::Drop;
 use std::ops::Index;
+use std::os::raw::c_void as std_c_void;
 use std::str::Utf8Error;
 
 ////////////////////////
@@ -844,9 +845,9 @@ pub struct Tensor<T: TensorType> {
     owned: bool,
 }
 
-unsafe extern "C" fn noop_deallocator(_: *mut c_void, _: size_t, _: *mut c_void) -> () {}
+unsafe extern "C" fn noop_deallocator(_: *mut std_c_void, _: size_t, _: *mut std_c_void) -> () {}
 
-unsafe extern "C" fn deallocator(_: *mut c_void, _: size_t, buffer: *mut c_void) -> () {
+unsafe extern "C" fn deallocator(_: *mut std_c_void, _: size_t, buffer: *mut std_c_void) -> () {
     tf::TF_DeleteBuffer(buffer as *mut tf::TF_Buffer);
 }