Update documentation to add the latest features

Author: csala

CONTRIBUTING.rst

@@ -69,8 +69,7 @@ Ready to contribute? Here's how to set up `MLBlocks` for local development.
 
     $ mkvirtualenv MLBlocks
     $ cd MLBlocks/
-    $ pip install -e .
-    $ pip install -r requirements_dev.txt
+    $ make install-develop
 
 4. Create a branch for local development::
 
@@ -88,7 +87,8 @@ Ready to contribute? Here's how to set up `MLBlocks` for local development.
 6. When you're done making changes, check that your changes pass flake8 and the
    tests, including testing other Python versions with tox::
 
-    $ make test-all
+    $ make lint       # Check code styling
+    $ make test-all   # Execute tests on all python versions
 
 7. Make also sure to include the necessary documentation in the code as docstrings following
    the `google docstring`_ style.

Makefile

@@ -130,7 +130,7 @@ view-docs: docs ## view docs in browser
 
 .PHONY: serve-docs
 serve-docs: view-docs ## compile the docs watching for changes
-	watchmedo shell-command -W -R -D -p '*.rst;*.md' -c '$(MAKE) -C docs html' .
+	watchmedo shell-command -W -R -D -p '*.rst;*.md' -c '$(MAKE) -C docs html' docs
 
 
 # RELEASE TARGETS

docs/advanced_usage/pipelines.rst

@@ -154,11 +154,11 @@ call is issued would be:
 
         X -> X1;
         X1 -> b1 [constraint=false];
-        b1 -> X2 [label=modified];
-        X2 -> b2 [constraint=false]
-        b2 -> X3 [label=modified];
-        X3 -> b3 [constraint=false]
-        b3 -> y
+        b1 -> X2;
+        X2 -> b2 [constraint=false];
+        b2 -> X3;
+        X3 -> b3 [constraint=false];
+        b3 -> y;
     }
 
 Another schema with some more complexity would be one where there is one primitive that
@@ -191,9 +191,10 @@ of actions would be:
             b1 -> b2 [style=invis];
 
             subgraph cluster_1 {
-                X1 [label=X];
-                f1 [label=features];
-                X2 [label=X];
+                {rank=same X1 f1}
+                X1 [label=X group=c];
+                f1 [label=features group=c];
+                X2 [label=X group=c];
                 f1 -> X1 [style=invis];
                 X1 -> X2 [style=dashed];
                 label = "Context";
@@ -204,8 +205,9 @@ of actions would be:
         {rank=same X features}
         features -> f1;
         X -> X1;
-        {X1 f1} -> b1 [constraint=false];
-        b1 -> X2 [label=encoded];
+        X1 -> b1 [constraint=false];
+        f1 -> b1 [constraint=false];
+        b1 -> X2;
         X2 -> b2 [constraint=false]
         b2 -> y
     }
@@ -242,9 +244,9 @@ do its job:
             b0 -> b1 -> b2 [style=invis];
 
             subgraph cluster_1 {
-                X1 [label=X];
-                f1 [label=features];
-                X2 [label=X];
+                X1 [label=X group=c];
+                f1 [label=features group=c];
+                X2 [label=X group=c];
                 X1 -> f1 -> X2 [style=invis];
                 X1 -> X2 [style=dashed];
                 label = "Context";
@@ -256,12 +258,92 @@ do its job:
         X1 -> b0 [constraint=false];
         b0 -> f1;
         {X1 f1} -> b1 [constraint=false];
-        b1 -> X2 [label=encoded];
+        b1 -> X2;
         X2 -> b2 [constraint=false]
         b2 -> y
     }
 
 
+JSON Annotations
+----------------
+
+Like primitives, Pipelines can also be annotated and stored as dicts or JSON files that contain
+the different arguments expected by the ``MLPipeline`` class, as well as the set hyperparameters
+and tunable hyperparameters.
+
+Representing a  Pipeline as a dict
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The dict representation of an Pipeline can be obtained directly from an ``MLPipeline`` instance,
+by calling its ``to_dict`` method.
+
+.. ipython:: python
+
+    pipeline.to_dict()
+
+Notice how the dict includes all the arguments that used when we created the ``MLPipeline``,
+as well as the hyperparameters that the pipeline is currently using and the complete specification
+of the tunable hypeparameters.
+
+If we want to directly store the dict as a JSON we can do so by calling the ``save`` method
+with the path of the JSON file to create.
+
+.. ipython:: python
+
+    pipeline.save('pipeline.json')
+
+Loading a Pipeline from a dict
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Similarly, once the we have a dict specification, we can load the Pipeline directly from it
+by calling the ``MLPipeline.from_dict`` method.
+
+Bear in mind that the hyperparameter values and tunable ranges will be taken from the dict.
+This means that if we want to tweak the tunable hyperparameters to adjust it to a specific
+problem or dataset, we can do that directly on our dict representation.
+
+.. ipython:: python
+
+    pipeline_dict = {
+        "primitives": [
+            "sklearn.preprocessing.StandardScaler",
+            "sklearn.ensemble.RandomForestClassifier"
+        ],
+        "hyperparameters": {
+            "sklearn.ensemble.RandomForestClassifier#1": {
+                "n_jobs": -1,
+                "n_estimators": 100,
+                "max_depth": 5,
+            }
+        },
+        "tunable_hyperparameters": {
+            "sklearn.ensemble.RandomForestClassifier#1": {
+                "max_depth": {
+                    "type": "int",
+                    "default": 10,
+                    "range": [
+                        1,
+                        30
+                    ]
+                }
+            }
+        }
+    }
+    pipeline = MLPipeline.from_dict(pipeline_dict)
+    pipeline.get_hyperparameters()
+    pipeline.get_tunable_hyperparameters()
+
+.. note:: Notice how we skipped many items in this last dict representation and only included
+    the parts that we want to be different than the default values. MLBlocks will figure out
+    the rest of the elements directly from the primitive annotations on its own!
+
+Like with the ``save`` method, the **MLPipeline** class offers a convenience ``load`` method
+that allows loading the pipeline directly from a JSON file:
+
+.. ipython:: python
+
+    pipeline = MLPipeline.load('pipeline.json')
+
 .. _API Reference: ../api_reference.html
 .. _primitives: ../primitives.html
 .. _mlblocks.MLPipeline: ../api_reference.html#mlblocks.MLPipeline

docs/conf.py

@@ -25,11 +25,11 @@
 from recommonmark.parser import CommonMarkParser
 # from recommonmark.transform import AutoStructify
 
-sys.path.insert(0, os.path.abspath('..'))
+# sys.path.insert(0, os.path.abspath('..'))
 
 import mlblocks
-
-mlblocks.add_primitives_path('../mlblocks_primitives')
+# 
+# mlblocks.add_primitives_path('../mlblocks_primitives')
 
 # -- General configuration ---------------------------------------------
 

docs/getting_started/install.rst

@@ -27,50 +27,23 @@ You can either clone the public repository:
 
 .. code-block:: console
 
-    git clone git://github.com/HDI-Project/mlblocks
+    git clone git://github.com/HDI-Project/MLBlocks
 
 Or download the `tarball`_:
 
 .. code-block:: console
 
-    curl  -OL https://github.com/HDI-Project/mlblocks/tarball/master
+    curl  -OL https://github.com/HDI-Project/MLBlocks/tarball/master
 
 Once you have a copy of the source, you can install it running the next command inside the
 project folder:
 
 .. code-block:: console
 
-    $ pip install .
+    $ make install
 
-.. _Github repo: https://github.com/HDI-Project/mlblocks
-.. _tarball: https://github.com/HDI-Project/mlblocks/tarball/master
-
-Additional Dependencies
------------------------
-
-The previous commands install the bare minimum requirements to make MLBlocks work, but
-additional dependencies should be installed in order to run the `quickstart`_ and various
-examples found in the documentation.
-
-The most important of these dependencies is the related project `MLPrimitives`_, which
-includes a huge list of primitives ready to be used by **MLBlocks**.
-
-Installing these additional dependencies can be achieved by running the command:
-
-.. code-block:: console
-
-    pip install mlblocks[demo]
-
-if **MLBlocks** was installed from PyPi, or:
-
-.. code-block:: console
-
-    pip install .[demo]
-
-if you installed **MLBlocks** from sources.
-
-.. _quickstart: quickstart.html
-.. _MLPrimitives: https://github.com/HDI-Project/MLPrimitives
+.. _Github repo: https://github.com/HDI-Project/MLBlocks
+.. _tarball: https://github.com/HDI-Project/MLBlocks/tarball/master
 
 Development
 -----------
@@ -81,4 +54,4 @@ order to be able to run the tests and build the documentation:
 
 .. code-block:: console
 
-    pip install -e .[dev]
+    make install-develop

docs/getting_started/quickstart.rst

@@ -24,8 +24,8 @@ them to the `MLPipeline class`_:
 
     from mlblocks import MLPipeline
     primitives = [
-        'sklearn.preprocessing.StandardScaler',
-        'xgboost.XGBClassifier'
+        'mlprimitives.feature_extraction.StringVectorizer',
+        'sklearn.ensemble.RandomForestClassifier',
     ]
     pipeline = MLPipeline(primitives)
 
@@ -34,8 +34,8 @@ Optionally, specific `hyperparameters`_ can be also set by specifying them in a
 .. ipython:: python
 
     hyperparameters = {
-        'xgboost.XGBClassifier': {
-            'learning_rate': 0.1
+        'sklearn.ensemble.RandomForestClassifier': {
+            'n_estimators': 100
         }
     }
     pipeline = MLPipeline(primitives, hyperparameters)
@@ -80,13 +80,13 @@ other ones will remain unmodified.
 .. ipython:: python
 
     new_hyperparameters = {
-        'xgboost.XGBClassifier#1': {
-            'max_depth': 10
+        'sklearn.ensemble.RandomForestClassifier#1': {
+            'max_depth': 15
         }
     }
     pipeline.set_hyperparameters(new_hyperparameters)
     hyperparameters = pipeline.get_hyperparameters()
-    hyperparameters['xgboost.XGBClassifier#1']['max_depth']
+    hyperparameters['sklearn.ensemble.RandomForestClassifier#1']['max_depth']
 
 Making predictions
 ------------------
@@ -99,8 +99,8 @@ labels.
 
 .. ipython:: python
 
-    from mlblocks.datasets import load_iris
-    dataset = load_iris()
+    from mlblocks.datasets import load_personae
+    dataset = load_personae()
     X_train, X_test, y_train, y_test = dataset.get_splits(1)
     pipeline.fit(X_train, y_train)
 

docs/index.rst

@@ -62,7 +62,8 @@ integrate with deep learning libraries.
    :caption: Pipeline Examples
    :maxdepth: 1
 
-   pipeline_examples/tabular
+   pipeline_examples/single_table
+   pipeline_examples/multi_table
    pipeline_examples/text
    pipeline_examples/image
    pipeline_examples/graph