parameter :-

All the values that we set constant for our neural network are called paramers for example, we train a network and then using back propagation we drive the weight matrix and bias matrix. These final-trained values are set to constant and then the model is ready for testing.

hyper-parameter :-

They both sound similar, but hyper-parameters are chosen prior to the assignment of parameter values. That means we first train our network to find out how many layers we need, how many neurons per each layer, drop out ratio, etc. Then using these optimized hyper-parameter values we train our model with back propagation for parameter setting. Now this hyper-parameter optimized model is best among its family of ensemble models.

EG.

learning rate

regularizers

strength of those regularizers

dimensionality of any hidden representations

number of decision trees (for random forest )

optimization algorithm (even could be)

• Hyper-parameters selection has significant impact on model performance.

Open-source hyper-parameter optimization packages,

1) Hyperopt -

explores hyper-parameter space strategically using “tree of Parzen estimators” (bayesian approach)

2) scikit-optimize -

new package

has 3 algorithms :-

1. forest_minimize(decision tree regression search)

2. gbrt_minimize (gradient-boosted-tree regression search)

3. gp_minimize (Gaussian Process Regression Search)

Above 2 are most prominent methods.

3) Spearmint -

under active development (license does not permit for commercial usage)

4) MOE -

growing as start-up

These packages, (benefit)

1. can set up very large space of hyper-parameters to explore

2. can reduce the number of choices (heuristically made)

• We compare the packages based on balance accuracy && speed. (test scores)
• The above both packages are based on Sequential model-based optimization.( at start samples small number of points from full hyper-parameter space, then run evaluations based on them)

5) Optuna

Baseline Method :-

These are the conventional optimization methods, on which we couldn’t find any improvements on qualities. Here we take 2 baseline methods to compare with our 2 packages.

1. Grid Search -

• exhaustive search (will try every combination of each settings of hyper-parameters)
• is guaranteed to find the best set of values(discrete version) in search space.
• not tractable for large parameter space

2. Randomized Search -

• samples from full grid search
• doesn’t offer the same guarantee as grid search
• extremely effective in practice

Packages -

• Basis for search is “dictionary mapping parameter names to values”
• we give a set of values for each key word. So this has one-to-many dictionary format.

EG . for baseline methods

param_grid = {

‘max_depth’ : [4, 8, 12],

‘learning_rate’ : [0.01, 0.3, 0.5],

‘n_estimators’ : [20, 50, 200],

‘objective’ : [‘multi:softprob’],

‘gamma’ : [0, 0.25, 0.5],

‘min_child_weight’ : [1, 3, 5],

‘subsample’ : [0.1, 0.5, 1],

‘colsample_bytree’ : [0.1, 0.5, 1]}

• We provide full list of values for dict. The algorithm only search among them.
• Mostly, the model expands for large dataset or architecture. So dict expands to very large list. So the algorithms have to search for each combinations including cross-combinations. So grid search is not tractable.

In Hyperopt, values are hyper-opt functions specifying distributions over parameter values.

Search is then samples the distributions over the each values.

hyperopt_grid = {

‘max_depth’ : hp.choice(‘max_depth’, range(4, 13, 1)),

‘learning_rate’ : hp.quniform(‘learning_rate’, 0.01, 0.5, 0.01),

‘n_estimators’ : hp.choice(‘n_estimators’, range(20, 205, 5)),

‘objective’ : ‘multi:softprob’,

‘gamma’ : hp.quniform(‘gamma’, 0, 0.50, 0.01),

‘min_child_weight’ : hp.quniform(‘min_child_weight’, 1, 5, 1),

‘subsample’ : hp.quniform(‘subsample’, 0.1, 1, 0.01),

‘colsample_bytree’ : hp.quniform(‘colsample_bytree’, 0.1, 1.0, 0.01)}

scikit-optimize only requires to specify the upper && lower bound for the search space.

scikit-opt wants lists of lists of values, rather than just dict.

So hard to keep track of which parameters are being specified.

The packge code handles internal conversions for dict — -> list && list — -> dict.

skopt_grid = {

‘max_depth’: (4, 12),

‘learning_rate’: (0.01, 0.5),

‘n_estimators’: (20, 200),

‘objective’ : Categorical((‘multi:softprob’,)),

‘gamma’: (0, 0.5),

‘min_child_weight’: (1, 5),

‘subsample’: (0.1, 1),

‘colsample_bytree’: (0.1, 1)}

Experimental Set-up :-

This will make max use of available data (whether its small or large)

1 ) Outer five-fold cross-validation

the full dataset is divided into five sections (folds). This creates five experimental settings, in which we train on four of the folds (merged together) and test on the fifth.

2 ) Inner five-fold cross-validation

For each experimental setting, the training set is used to find the optimal hyperparameters — for each setting of the hyperparameters, a full round of five-fold cross-validation is performed using only the training data.

3 ) hybrid

The hyperparameter setting that does best at step 2 is the basis for the model that is used for testing in the outer cross-validation.

For each (train_folds, test_fold) in the outer cross-validation:

For each setting S of the hyperparameters:

Cross-validate S on train_folds

Let S* be the setting that did best in the inner loop.

Use S* to assess on test_fold.

• Here if we considered the parallelization, then the performance for each packages and baseline will be different.

We can segment and provide data sets to each algorithms for better performance.

full_experimental_run = [

(grid_search, param_grid, {}),

(random_search, param_grid, {‘sampsize’: None}),

(hyperopt_search, hyperopt_grid, {‘max_evals’: 100}),

(skopt_gp_search, skopt_grid, {‘n_calls’: 100}),

(skopt_forest_search, skopt_grid, {‘n_calls’: 100}),

(skopt_gbrt_search, skopt_grid, {‘n_calls’: 100})]

• Usually, from experiments the the time taken is,(in fast measurement)

2 packages > > grid search

2 packages > random_search

• Random search strongly shows the brute-force approach.

Experiment should be conducted on cases of,

Small datasets

Small features, but large observations

large features, but small observations

Large datasets

Summary :-

— No big difference between Hyperopt && scikit-opt. They consistently perform better && deliver more effective models faster && easy to use.

— But clearly better than baseline approaches.

— These baseline approaches needlessly expensive.

References :-

1. https://roamanalytics.com/2016/09/15/optimizing-the-hyperparameter-of-which-hyperparameter-optimizer-to-use/
2. http://www.cs.ubc.ca/~hutter/papers/11-LION5-SMAC.pdf
3. MOE — https://github.com/Yelp/MOE

⁃ In Single Perceptron / Multi-layer Perceptron(MLP), we only have linear separability because they are composed of input and output layers(some hidden layers in MLP)

⁃ For example, AND, OR functions are linearly-separable & XOR function is not linearly separable.

⁃ We atleast need one hidden layer to derive a non-linearity separation.

⁃ Our RBNN what it does is, it transforms the input signal into another form, which can be then feed into the network to get linear separability.

⁃ RBNN is structurally same as perceptron(MLP).

⁃ RBNN is composed of input, hidden, and output layer. RBNN is strictly limited to have exactly one hidden layer. We call this hidden layer as feature vector.

⁃ RBNN increases dimenion of feature vector.

⁃ We apply non-linear transfer function to the feature vector before we go for classification problem.

⁃ When we increase the dimension of the feature vector, the linear separability of feature vector increases.

A non-linearity separable problem(pattern classification problem) is highly separable in high dimensional space than it is in low dimensional space.

[Cover’s Theorem]

⁃ What is a Radial Basis Function ?

⁃ we define a receptor = t

⁃ we draw confrontal maps around the receptor.

⁃ Gaussian Functions are generally used for Radian Basis Function(confrontal mapping). So we define the radial distance r = ||x- t||.

Gaussian Radial Function :=

ϕ(r) = exp (- r²/2σ²)

where σ > 0

⁃ Example. XOR function :-

⁃ I have 4 inputs and I will not increase dimension at the feature vector here. So I will select 2 receptors here. For each transformation function ϕ(x), we will have each receptors t.

⁃ Now consider the RBNN architecture,

⁃ P := # of input features/ values.

⁃ M = # of transformed vector dimensions (hidden layer width). So M ≥ P usually be.

⁃ Each node in the hidden layer, performs a set of non-linear radian basis function.

⁃ Output C will remains the same as for the classification problems(certain number of class labels as predefined).

⁃ Only Nodes in the hidden layer perform the radian basis transformation function.

⁃ Output layer performs the linear combination of the outputs of the hidden layer to give a final probabilistic value at the output layer.

⁃ So the classification is only done only @ (hidden layer → output layer)

Training the RBNN :-

⁃ First, we should train the hidden layer using back propagation.

⁃ Neural Network training(back propagation) is a curve fitting method. It fits a non-linear curve during the training phase. It runs through stochastic approximation, which we call the back propagation.

⁃ For each of the node in the hidden layer, we have to find t(receptors) & the variance (σ)[variance — the spread of the radial basis function]

⁃ On the second training phase, we have to update the weighting vectors between hidden layers & output layers.

⁃ In hidden layers, each node represents each transformation basis function. Any of the function could satisfy the non-linear separability OR even combination of set of functions could satisfy the non-linear separability.

⁃ So in our hidden layer transformation, all the non-linearity terms are included. Say like X² + Y² + 5XY ; its all included in a hyper-surface equation(X & Y are inputs).

⁃ Therefore, the first stage of training is done by clustering algorithm. We define the number of cluster centers we need. And by clustering algorithm, we compute the cluster centers, which then is assigned as the receptors for each hidden neurons.

⁃ I have to cluster N samples or observations into M clusters (N > M).

⁃ So the output “clusters” are the “receptors”.

⁃ for each receptors, I can find the variance as “the squared sum of the distances between the respective receptor & the each cluster nearest samples” := 1/N * ||X — t||²

⁃ The interpretation of the first training phase is that the “feature vector is projected onto the transformed space”.

Advantages of using RBNN than the MLP :-

1. Training in RBNN is faster than in Multi-layer Perceptron (MLP) → takes many interactions in MLP.

2. We can easily interpret what is the meaning / function of the each node in hidden layer of the RBNN. This is difficult in MLP.

3. (what should be the # of nodes in hidden layer & the # of hidden layers) this parameterization is difficult in MLP. But this is not found in RBNN.

4. Classification will take more time in RBNN than MLP.

Radial Basis Functions Neural Networks — All we need to know was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

>>uses>> Decision Tree Model for Parameterization

>>integrates>> 3 techniques,

1. a Sampling Technique,

2. a Subspace Method, and

3. an Ensemble Approach

to optimize the model building

** sampling approach** := bootstrap; adopts a Random Sampling Approach with Replacement.

** subspace method** := also adopts a Random Sampling approach,

but helps extract smaller subsets(subspaces) of features.

helps to build decision tree based on them.

select decision trees for the random forest construction.

** ensemble approach **:= based on bagging approah; helps to build classifiers.

1.1 Random Forest(RF)

Tress can be

1. classification tress

2. regression trees

So RF can be applied to both.

• decision tree provides single trained decision tree classifier for the testing phase.
• But RF provides multiple trained DT classifiers for the testing phase.
• This property makes RF more preferred over regular DT learning.

1.1.1 Parallel Structure

• Parallelization is one of the contributing properties of the RF toward the “enhanced classification performances”.
• DT can be built at the same time for classssification for the RF modeling.
• This RF parallel structure can help for Big Data Classification.

1.1.2 Model Parameters

• Parameterization of RF includes the parameterization of a DT model becasue it adopts the other one inside.
• Therefore, RF model consists of the parameters of DT, but has some more new parameters too.

eg. # of feature subsets, selected for the # of DTs for the construction of RF

1.1.3 Gain/Loss Function

• parameters are optimized based on

1. bootstrap sampling

2. subspace integration

in the model building

• Tree building uses the same quantitative measures as used in DTs to select best features, eg.

1. Entropy

2. Gini Impurity

3. Information Gain

1.1.4 Bootstrapping & Bagging

• for optimizing the classification objectives,

1. statistical measures,

2. bootstrapping,

3. bagging

plays important roles in RF learning.

1.1.4.1 Bootstrapping (BS)

• BS is a simple randomization technique.
• Its effect in RF is magnificent.
• It selects several subsets from a set of data by randomly selecting, but totally the same # of observations as in the original data set, but with replacement.
• So some observations can be appeared many times in a subset of the data set.
• Notion :-

maximize the “class-distance” at each intermediate node & the leaves of the DT.

• BS is applied @ the training phase of the RF algorithm.

1.1.4.2 Overlap Thinning

• BS brings some effects to a data domain.
• BS generates several domains where the “classes” may be “isolated” more than the original data domain.
• The classes in these isolated data domains can be easily separated by single splits.
• Sometimes we may not need splitting.

1.1.4.3 Bagging

• means the averaging of the prediction(classification) responses, the BS samples gave, to obtain final prediction result.
• Bagging came from Bootstrap Aggregating.
• BS creates multiple domains with “simple class overlap thinning”, these domains help create multiple classification models, and the multiple classification models allow the testing algorithm to evaluate the performance of the classifiers efficiently.
• Bagging is applied at the testing phase of the RF algorithm.

1.2 Random Forest Learning Model

• achieves parameterization objectives & optimization objectives.

1.2.1 Parameterization

• parameters,

1. # of domains generated using bootstrapping — A.

2. size of the subspace for each node(domain) — B.

3. threshold for a domain split — C.

• Goal of parameterization:-

we need to find or suggest possible values for the parameters.

• usual value for A is 10.

But it is appropriate to conduct “cross-validation” and determine a suitable range of values.

• for the B selection, there is 2 constraints.
• first recommendation is n << p & it must be kept contant throughout the construction of DT.
• second constriant, n <= sqrt(p); we have p features in the original data domain. n of them are good ones. Fraction of good ones in n/p. Among them one of the feature is best one, its fraction among good ones is 1/n.

• people use sqrt(p) as the # of features for random selection.

1.2.2 Optimization

once the parameters are selected, the model is parameterized. Then the parameters must be optimized.

• But in RF parameterization & optimization are nested, & they are performed at the same time.
• Optimization has been done at each node of the tree.
• Major players for the RF optimization are the Gini index(entropy) & the information gain.

1.3 Random Forest Learning Algorithm

• consists 3 parts as any other machine learning algorithms; training, validation, and testing.
• Cross-validation is already included in the training agorithm.
• So no separate validation algorithm is needed for RF technique.

SIMPLY :-

RF learning gives us several spaces (best feature combinations), where the best domain divisions can be performed & high classification accuracies can be obtained.

• Therefore, RF classification objective is divided into only training & testing algorithms.

1.3.1 Training Algorithm

• This provides a systematic approach to develop multiple classifiers(DTs) so that testing algorithm uses multiple classifiers to select the best way to classify the new data.

Step 1 :: —

— multiple subspaces are created for given data.

r <= sqrt(p)

p = # of features ;

r = dim of subspaces

Step 2 :: —

— we have significantly smaller r (dimensional subspace), so we can find the best feature & best split location(domain division) for the root node using the DT building process.

[But we DO NOT perform this process to this subspace, so skip to next step]

Step 3 :: —

— alter the subspace randomly using bootstrap samples & create multiple bootstrap-subspaces, where the overlap thinning occured in data domains.

Step 4 :: —

— construct nodes for the DTs by applying DT algorithm to find best feature & the best split location for each bootstrap-subspace.

Step 5 :: —

— Now we have a DT(classifier) for each bootstrap sample.

— These trees can be used by the testing algorithm to classify the new data.

Advantages of RF over DT,

1. multiple DTs are available for classification at the test space => will increase the classification accuracy.

2. bootstrap samples are used for multiple DTs, which will help increase the “sharpening” of the classification boundary through overlap thinning.

3. the exhaustive search done for finding the best feature & best split location in the entire space is eliminated by the subspace search => small additional computational cost in bootstrap sampling. (Computational Advantages)

1.3.2 Testing Algorithm

• Testing requires the entire RF can be constructed using several bootstrap samples(10) in early. So then the labelling process can be started.
• Steps,

1. Feature Selection :

the correct sequence of features from the incoming data for the nodes of the trees must be “observed”.

2. Tree Selection :

we have N number of RF classifiers. So we push the new data according to the features through all of the DTs & obtain its class labels.

3. Bagging :

trees selected in the previous step are used to find the aggregate of the results obtained (bagging technique)

Finally, we can conclude the new observation belongs to a specific label by voting mechanism.

Random Forest Scalability

• In typical data science application,

# of features is fixed.

RF technique selects a subspace from them & builds the “depth-first decision trees” based on bootstrapping to construct RF.

• But in Big Data,

feature set can dynamically “grow”.

• The problem can be handled by building “breadth-first decision trees”. Here the DTs were build in parallel.

However, in depth-first & conventional RF, the DTs are build one after the other DT is completely constructed.

Reference :-

Machine Learning Models and Algorithms for Big Data Classification, Shan Suthaharan; Chapter 11 — Random Forest Learning

Random Forest Learning-Essential Understanding was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Frequent terminologies,

Data Domain := node

Events := Observations

Subdomains := leaves of current tree

• Hierarchical supervised learning models.
• Main hierarchical model is DECISION TREE.
• It has 2 categories,

1. Classification Tree

2. Regression Tree

for building decision tree — — > tree split algorithms

for building efficient tree via training — — > requires quantitative measures

• These measures are,

1. entropy

2. cross-entropy

3. Gini impurity

4. Information Gain

• Discussion of “training algorithms” is suitable for both classification & regression trees.”

Decision Tree

• Defined as “rule-based, binary-tree building technique.”
• But easier to INTERPRET as a “hierarchical domain division technique.”
• Is a supervised learning model, which HIERARCHICALLY “maps” a DATA DOMAIN onto a RESPONSE SET.
• Divides a data domain(node) recursively into 2 sub domains(have HIGHER information gain) THAN was before split.
• GOAL of SUPERVISED learning ====> Classification of Data
• Information Gain — — means — → EASE of classification in the subdomains (created by split).
• Optimization Algorithms — -GOAL — — -> find the best split (giving max infromation gain).

Data Domain := D (in sorted list)

Class Lables := L (corresponding)

Information Gain := Ii

The 2 Sub Domains = D1, D2

The 2 Domain Labels = L1, L2

Best split = m

• Spread (or Distribution Pattern) over the class labels → determines the ease of classification.
• If m is considered as the best split,

average (Ii1, Ii2) > Ii

Here Ii1, Ii2 are the information gains of the 2 subdomains derived after first previous split.

• We need a GOOD qualitative measure for measuring Information Gain.
• p0, p1 => probabilities class 0, 1 drawn from the domain D.

|p0 — p1| — — > 1

— one particular class DOMINATES highly in that domain.

— domain DIVISION is NOT required.

On the other hand,

|p0 — p1| — — > 0

class have equal domination in that domain

further split is needed

1)

• q0, q1 => probabilities class 0, 1 drawn from subdomain D1.
• If our Split is Efficient,

q0 > p0

OR

q1 > p1

2)

3)

— The absolute differences are QUANTITATIVE MEASURES that measures PROPORTIONALITY between the classes in respective SUBdomains.

Types of Decision Trees

• Classification Decision Tree — — > predict a class label (discrete)
• Regression Decision Tree — — > helps predict a value (continuous)

for a response variable

Classification Tree

• We need to construct the tree at the training phase.
• The propagation of tree split divides data domain.
• “generation” of SIMPLE, MULTIPLE, THIN layers of data domains consists of SPLIT-REGIONS.
• The classified subdomains should be differnet “Disjoint Subdomains”.
• Main Parameters,

— — “values of the features” used to split tree at a particular node.

— — these parameters TRAINED using INFORMATION GAIN as quantitative measure.

— — combination of selected parameters — — > classifier

• Subdomains(leaves of current tree) also have “Class information”
• “split criterion” needs “decision parameters” that form the classification model.
• Decision parameters := a feature & its split location

— — — — — by maximizing the information gain

Regression Tree

• It helps to assign a value for new data.
• It divides data domain into disjoint, rectangular subdomains by “splitting features”, & MAPS the subdomains to NON-EVERLAPPING groups of RESPONSES that are ESTIMATED with minimum ERROR criterion.
• Method has 4 tasks,

1. feature selection for a node.

2. parameterization of the split location.

3. parameterization of the depth of tree

4. estimation of the response variables.

Decision Tree Learning Model

• Modeling — -means — -> definition of a function Or MAPPING between a DATADOMAIN and a RESPONSE set, followed by PARAMETERIZATION of the model & the OPTIMIZATION of the parameters.

Parameterization :-

— a “supervised model” must be PARAMETERIZED so that it can be TRAINED, VALIDATED, & TESTED (This is true for decision tree too).

— how to parameterize tree like structure? how to find the parameters?

— In parameter building,

-nodes are split,

-leaves are generated

recursively.

— we can parameterize,

1. the coise of features for a node.

2. the threshold that splits the node and divides the feature set into 2 subsets.

3. the number of levels that the entire tree should have.

[4. features.]

— parameterization :-

*parameter1* select the features for the root node & intermediate nodes.

1. Can be chosen randomly

2. May select a feature that gives max information gain/Error Reduction

*parameter2* parameter for splitting criterion,

1. may select the “statistical techniques; mean/ median” to find the split location

2. find split location, that give an accepatble information gain.

*parameter3* select a parameter to STOP tree BUILDING. ==> # of levels in decision tree := is a parameter.

— Optimization :-

** select an ALGORITHMS that helps to optimize the parameters ==> final decision tree is OPTIMAL ==> tree can perform very good PREDICTION

** may lead to a computationally expensive process.

OPTIMIZATION :-

• divides decision tree EFFICIENTLY in training.
• This determines which feature must be selected FIRST to start building the tree?

Which feature must be selected @ INTERMEDIATE steps of tree building?

• Therefore, we need a GOOD Quantitative Measure
• Tree building — -means — -> generating LEAVES/ BRANCHES from a node, by carrying out the algorithm ITERATIVELY.

Quantitative MEASURES

• We need Quantitative measures in 2 places,

1. to measure information gain while splitting

2. to measure SIGNIFICANCE in the class difference in each node to decide further split.

• For a useful measures, the requirements are,

1. first requirements

1. entropy

2. Gini impurity

3. information gain

2. second requirement

1. class proportions

2. count differences

3. probability measures (ratios & percentages)

ENTROPY & CROSS-ENTROPY

• based on “proportionality of the events.”
• if one event occurs more than another event in a PLACE or with an OBJECT,

then we HAVE A GOOD KNOWLEDGE about that PLACE or OBJECT relative to the MAJORITY event.

• if both events occur the same NUMEBR of times in that PLACE or with the OBEJCT, then it is HARD to Characterize.

• This graph is plot against probability index VS entropy

Gini Impurity

• used to measure INCORRECT labelling with MATCHING PATTERNS.
• Gini — means — -> “generalized inequlity index”
• may be used to build decision tree instead of entropy

• differences in graphy of Gini and entropy

1. spread of plot

2. skewness (symmetry)

Entropy ====> useful, when “multiple events occur at a PARTICULAR instance or a location”

Cross-Entropy ====> useful, when “multiple events(but the same events) occur at 2 different instances OR locations.

Entropy =====> events come from SAME STATISTICAL distribution

X-Entropy =====> events are from 2 different distributions.

entropy within a distribution => intra-error

Cross-Entropy => inter-error

Information Gain

• binary events
• entropy OR Gini used to “Describe these Characteristics.”

Decision Tree Learning Algorithm

• GOAL :-

* to focus on EACH one-DIMENSIONAL subspace(single feature at a time),

* select the best feature(best one-dimensional subspace) and BEST split LOCATION,

* extract the feature value at the best split location,

* divide the domain into 2 subdomains.

REPEAT until **NO** Domain divisions are required

Training Algorithm

• first & most important requirement => best feature & best split location
• parameterization & optimization are PERFORMED “simultaneously” @ EACH node WHILE a decision tree is BUILT.

Reference :-

Machine Learning Models and Algorithms for Big Data Classification, Shan Suthaharan; Chapter 10 — Decision Tree Learning

Virtual-Environment is a tool that creates an isolated environment in Python. When we download and install virtualenv inside a desired folder, a bin sub-directory is created and it has all the necessary executables to support python execution.

We can create many instances of virtual environment in that folder by executing the command with the desired instance name. However, the disadvantage it has is that we cannot transfer the folder to different machine or platform. That means virtualenv is not portable.

When the virtualenv instance is in run mode, we can execute the regular python package manager commands like pip and python commands.

To begin, using the virtual environment, it needs to be activated:

$ source my_project/bin/activate

Then install the packages as usual, for example:

$ pip install requests

If we are done working inside the virtual environment for the moment, we can deactivate it:

$ deactivate

This puts us back to the system’s default Python interpreter with all its installed gloabl libraries.

To delete a virtual environment, just delete its folder. (In this case, it would be rm -rf my_project.)

virtualenv — wrapper

virtualenv wrapper provides a set of commands which makes working with virtual environments much more pleasant. It also places all your virtual environments in one place.

$ pip search Pympler

SEARCH A PIP PACKAGE NAME

$ pip install Pympler

INSTALLING A PIP PACKAGE NAME

$ pip list

LIST OUT THE PIP PACKAGES INSTALLED CURRENTLY

$ pip uninstall Pypmpler

UNINSTALL THE NAMED PIP PACKAGE

$ pip list — outdated

ONLY LIST THE OUTDATED PIP PACKAGE

OR

$ pip list -o (small o letter)

$ pip install -U setuptools (Capital U Letter) → Update

— — * — — * — — * —EXPORTING PIP PACKAGES— — * — — * — — * —

$ pip freeze

This will output all currently installed pip packages ( into a requirements.txt file)

$ pip freeze > requirements.txt

$ cat requirements.txt

This will show the file contents

— — * — — * — — * — IMPORTING PIP PACKAGES — — * — — * — — * —

$ cat r_test.txt

$ pip install -r r_rest.txt

This tells that we gonna use a requirement file

$ pip list

$ pip freeze- -local

— -local → will not allow us to access global properties

-nl (flag) → tells that one at a time

— —* — — *— —* — —* — — *— — * — — * — — * — — * — — * — — * —

Pip for virtual environment

$ pip install virtualenv

$ pip list

$ mkdir Environments

$ cd Environments

$ virtualenv project1_env

$ source project1_env/bin/activate

> From this last command, the prompt will begins with virtual env name now onwards

# which python

# which pip

# pip list

This will list only local packages, not global

# pip install bumpy

# pip install scipy

# pip list

# pip freeze — local > requirements.txt

This will export only the local entities

# cat requirements.txt

# deactivate

$ which python

$ pip list

$ ls

> This commands will show only 2 files according to what we typed in cmd. requirements.txt AND project1_env

$ rm -rf project1_env

$ ls

> This will show only the requirements.txt because the project folder was deleted.

NOW INSIDE PYTHON 2.6 VERSION CODE (This tells that we can initialize virtualenv even inside the lib python folder too)

$ virtualenv -p usr/bin/python2.7 py26_env

$ ls

$ source active py26_env

# which python

# ls

# pip install -r requirements.txt

# deactivate

Python modular import has 2 options,

1. Absolute modular imports

2. Relative modular imports

• Absolute imports exist from the beginning of python release; however, relative import came once after python 2.7.
• Relative imports use module’s name attribute to determine the module’s position in the package hierarchy.

A project with multiple hierarchical level directories work as a package with some attributes.

1. There should be a __init__.py file(usually kept empty) in a directory. When a python scripts need to look down into the hierarchical level, the existance of __init__.py file will lead the program flow. Otherwise “package not found error” occurs.

2. The script should be called while running by standing out from the package.

python -m package.subpackage.subsubpackage.pyscript

We can run python scripts in 2 ways.

1.We step into the python script directory and call python mypy.py [Current script is a top-level module]

• In this case, the import module canot step out of the directory in hierarchical tree.
• Here code runs on global namespace.

2. We stand ahead of the python package and call python -m myPackage.subPackage.subsubPackage.myScript [modular level module]

• Here -m tells the modular initialization.
• We drop .py at the end of script extension.
• We direct the script to run by the use of namesapces.
• Namespapces are just the names given to each directory in the package.
• This script run call tells exactly where the module is actually located inside the package.
• Here the code runs as a part of an imported module.
• As this code runs not in global namespace, the __name__ becomes the name of the module.

Namespace for python script :-

Usually, say for example if we have a python script called myPython.py, when we run this script, a namespace will be added to each module, and by default it is __main__.

def funcPy():

pass

if __name__ == ‘__main__’:

print ‘__main__’

else:

print __name__

So basically, python has name module, which holds all namesapces inside. By deafult when a script runs, it grabs the global namespace __main__. We can call this name of the script by __name__ variable. Samelike, we can get package name __package__ and directoty name __dir__. If a script has multiple import and from statements, those modules will take their file name as namesapces, but our primary running script will take __main__ namesapce. However, we can manually change this name. One way is to change in /python2.7/lib/__init__.py folder and other way is to re-initialized inside the code.

By default, __package__ will be assigned to None & __name__ will be assigned to __main__.

******************Don’t Forget*****************

When we use relative modular imports, we shoud stay outside the package and call,

python -m myPackage.subPackage.subsubPackage.myScript

Otherwise, runtime error will rise.

“You’re not using it as a package.”

**********************************************

The __init__.py files are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as string, from unintentionally hiding valid modules that occur deeper on the module search path. In the simplest case, __init__.py can just be an empty file, but it can also execute initialization code for the package or set the __all__ variable, described later.

1. from packagePy import *

2. __all__ = [“module1”, “module2”, module2]

The example for the above implementation is,

PackagePy

1. module0

2. module1

3. module2

4. module3

5. module4

The difference between above 2 is that we don’t have to import all modules inside the package in (2)nd one.

Python searches a list of directories to resolve names while import statements. Because these names can be any directory, or arbitrary names can be added by the developer. However, the developers have to worry about directories that happen to share the same name with a valid Python module, such as ‘string’. To differentiate those 2 modules(default python lib module & user defined module), relative modular import comes into play. The creation of __init__.py will direct the script to search for the namespace in sub directories on hierarchical level of the package. If you remove the __init__.py file, Python will no longer look for submodules inside that directory, so attempts to import the module will fail.

Reference :-

1. https://stackoverflow.com/questions/37193670/valueerror-attempted-relative-import-in-non-package-not-for-tests-package
2. https://stackoverflow.com/questions/14132789/relative-imports-for-the-billionth-time

Docker Build file is a text file that creates our own custom docker image. Docker Build file builds docker image from docker file.

Examples here are done in Mac.

$ touch Dockerfile

— — — creates a Dockerfile

$ docker ps

— — — list of images, currently running. Usually nothing shows up as no docker images run on background.

$ docker ps -a

— — — lists down the previously ran docker images. Those docker images will still exist in the machine unless the user manually remove them once task completed.

$ docker run -ti -rm ubuntu /bin/sh

— — — When we use -rm in the command, the given docker image will be removed when exit the application. Here t denotes terminal & i points interactive mode.

Now the docker will start running & our terminal will change to docker image prompt mode(interactive mode changes from $ prompt to # prompt- depending on the OS used and current state).

# whoami

— — — will give root

# get - -version

# vim

# curl

# wget

— — — these above commands should be initialized in the docker file while creating the docker image incase we prefer to use them in our docker image.

# exit

— — — will exit the docker image run mode.

$ docker ps

— — — Now this command will show Nothing because we said to remove the image once we exit in the docker run command(No images of lastly opened)

DOCKER FILE

First line should start with ‘FROM’. We can import or start the docker image from scratch.

FROM ubuntu

MAINTAINER Ramraj@ramraj@emory.edu

MAINTAINER is not that important because we just use it to mention the create name and contact email id.

RUN is going to add some package managers inside

: RUN app update

: RUN app add vim

: RUN apk curl

$ ll

— — — will list down the docker files

$ docker build -t ramraj/ubuntu-custom:1.0 .

— — — Here -t denotes the tag, just the version. And ‘.’ tells that the current folder (usually path to folder) of the docker file.

STEPS are being created while building the docker file. It creates final docker image as well as intermediate docker images(with name “none”). After the completion of STEP 1/N (N lines in docker file), the created image will have an unique ID.

$ docker images

— — — will give the specified image with TAG 1.0

$ docker images -a

— — — will give too many images(includes intermediate images). But only the final docker image has the name and TAG. This scenario is called Image cache. Each lines in the docker file create each docker images for the purpose of caching. When we run the docker build again less number of steps will be displayed.

$ docker build -t ramraj/ubuntu-custom:1.0 .

— — — Using cache the building process will speed up.

$ docker run -ti -rm ramraj/ubuntu-custom:1.0 /bin/sh

Now if we changed a step in Docker build file, say for example

RUN: APK GIT from APK CURL

When we build the docker image again, the non-changed steps still run from cache. But the changed line is executed again. Here a docker image is created, called Dangling Image

$ dicker images — filter “dangling=true”

When we change mind and want to change the docker image, the previous image would be abandoned in the pool. That is called dangling image.

$ docker mi $(docker images -q — filter “dangling=true”)

— — — Here -q will remove the HEADING displayed.

If we break the docker building cycle, then the flow interrupts, and the cache flow will be reorganized. Each intermediate docker images, consumes almost same space. If the space is large for a docker image, then the whole steps of intermediate docker images consume a lot of spance.

RUN apk update && \

apk add curl && \

ask add vim && \

apk aadd git

Instead of creating many steps, includes all steps in one. Total numeber of steps is 3 now, NOT 5 (FROM, MAINTAINER, RUN) = OPTIMIZATION.

Usual Docker file initializers are FROM, MAINTAINER, COPY, ADD, LABELS, ENV, EXPOSE (for https), USER, WORKDIR, RUN.

Now we create a file named config.txt.

COPY ./config.txt /usr/src/config.txt

— — — Add this statement to Docker file.

$ docker build -t ramraj/ubuntu-custom:1.0 .

$ docker run -ti -rm ramraj/ubuntu-custom:1.0 /bin/sh

# ls /usr/src

— — — will return config.txt as the output display.

# cat /usr/src/config.txt

— — — will show what we have already written and saved on the config.txt file.

This change of adding a new file into the Docker build file, will too rebuild the docker images by non-caching intermediate docker images. Somehow docker understands the change made in its build flow.

Reference :-

https://www.youtube.com/watch?v=6Er8MAvTWlI

Hard Terms, known in advance :-

Gaussian Process

Noise-Free Global optimization

Noisy Global optimization

Acquisition functions

Surrogate Functions (/Gaussian Process)

Stochastic

Bayesian Regression

What is Gaussian Process ??

- It is an Bayesian Regression

- It is an exact interpolation regression method.

- It is a natural generalization of linear regression.

• It specifies a Distribution over functions.

Bayesian Optimization is a method that uses some kind of approximation. Imagine, if we don’t know a function, what we usually do? Ofcourse, we will try to guess or approximate it with some know prior knowledge. The same idea is behind the posteriori probability. The criteria here is that we have observations, where the data are coming records by records (online-learning), so we need to train this model. The trained model will obviously obey a function. That function that we don’t know will absolutely depend on the learnt data. So our task is to find the hyper-parameters that maximizes the learning schema.

We have some variables here. One is the observation records(features + labels) and the second is the parameters, which defines the model.

eg.

(say for like in y = mx + c ; m & c are parameters, and y, x are variables-labels & features. We have an obligation to find these m & c to learn our best model)

Bayesian Optimization helps to find a best model among many. We have cross-validation in hand. But how many samples we gonna try on a pre-list to choose a best model among them. That’s why Bayesian approach speed up the process by reducing the computation task and doesn’t expect help from the person to guess the values. This optimization technique is based on randomness and probability distributions.

To say in my words, the first observed point comes to the model. we draw a line. Then the next point follows. We joint those 2 points and draw an adjusted line from previous curve. A third point comes, so we draw a non-linear curve. When the number of observed points increases, the number of combination for possible curves increases. It is same like sampling theorm in Statistics where we estimate a population parameter from sample parameters.

We plot another histogram curve along with the non-linear objective function curve. That tells, where the max value will present among all past observations for a given objective function curve(finding arg max).

Our goal here is not to determine the objective function(which is unknow) as a curve completely with as much as observed points as possible. Instead, we need the arg of the maximum value(index of the max function). So we need to divert out concentration towards the histogram function. When the possible combination of objective function curves increases the histogram forms a distribution, which describes the acquisition funtion. It is the idea behind Bayesian Optimization.

REMEMBER, our goal, primarily need to determine the arg of the max value, and secondarily selecting next best possible max value, which could be on the function curve.

We see many waggle curves in the plot. That tells the range of the function curve next point could be located at given a sampled point. The variance gives the distribution from the mean value on the function curve.

When we estimate by superimposing the observed values over the unknown function curve, then this is noise-free optimization. But when we need to consider noisy optimization, our unknown function will slightly diviates the observed points by a noisy error value.

We define the acquisition function under many measures and some of them are here,

1. PMAX

2. IEMAX

3. MPI = Maximum Probabiity of Improvement

4. MEI = Maximum Expected of Improvement

5. UCB = Upper Confidence Bounds

6. GP-Hedge

MM = Maximum Mean

MUI = Maximum Upper Interval

Acquisition Criteria :- which sample should be sleceted next

Surrogate Function/ Response Surface(model)

• posterior over unobserved points based on prior.
• its parameters might be based on prior.

An Acquisition function :-

The measurement is Probability of Improvement(PI).

μ+ = the best observed value among the past points.

We are looking for a max value of the unknown function. But among our past observation, the next possible max value should be larger than the current max observed value.

ε = very small arbitary value denotes the point in Guassian Distribution.

Our Goal :-

PI(x) = The probability of the next max of the unknown function found to be above the max observed value plus some bias.

This is just the 1D integration of the Gussian distribution with the threshold boundary of the max observed value.

For the each point in the unknown function, we consider a Guassian distribution placed vertically at each point samples. This distribution has mean at the value on the unknow function & some certain variance.

We standardize this in the above Kushner 1964 equation.

Exploration & Exploitation are the frequently used terms to explain a phenomena and the optimization algorithm. There is a trade-off exists in-between these 2. Therefore, we balance this tradeoff using the “acquisition function”.

Our next chosen point( x ) should has high mean (exploitation) & high variance (exploration).

Because we search for the next point in high variance distribution. That means exploring the point x.

High mean means we chose the next point( x ) with high offset/ bias so that we should give some exploitation weights to decrease the mean of the next point.

The above picture gives some intuition about our algorithm. The red curve is our True Objective function curve. Unfortunately, we don’t know this function and its equation. So we gonna approximate it using Gaussian Process. Throughout our Sampled Points(given 4 samples here), we draw a intuitive/ confident curve that fits with our observed samples. So the green region shows the Confidence Region, where the most probability of curve points could be positioned. From the above prior knowledge, we determine that second point(f+) as the maximum observed value. So the next max point should be above it or equal. We draw a blue line here. The next max point should fall above this line. So from the intersecting points of f+ and Confidence Region, we could assume that the curve samples below the f+ should be discarded along with our goal to find arg max. Therefore, now we have narrowed down our area of investigation. This same process continues with the next Sampled Points.

Reference :-

Shallow Understanding on Bayesian Optimization was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Autoencoder is a neural network (NN), as well as an un-supervised learning (feature learning) algorithm.

• It applies backpropagation, by setting the target value same as input.
• It tries to predict x from x, without need for labels.

• It tries to learn an approximation of an “identity” function.
• But we place constraints on the network like less number of units in hidden layer.
• It represents orig input from compressed, noisy, or corrupted data.
• It consists of narrow hidden layer between Encoder & Decoder.

• Intermediate layer (middle layer[bottle neck] or latent space representation) is a compresssed representation of the input data.
• The difference in Autoencoder VS Engineered compression is,

— — — In Autoencoder compression & decompression functions learn from the data itself not like jpg and zip.

• Encoders and Decoders are feed-forward NN.
• The network is trained by minimizing the difference betweeen input and output.
• If the input is completely random(chosen from IID & independent from other features), then compression task would be difficult.
• If the input features are correlated, then the network will discover those.
• Its learning style is like PCA, so it closely relates to PCA(Principal Component Analysis).
• We can observe it from the similarity of Eigen vactor representation.
• The differences between PCA & Autoencoders similar

— autoencoders are much flexible than PCA.

— NN activation functions introduce “non-linearities” in encoding, but PCA only linear transformation.

— we can stack autoencoders to form a deep autoencoder network.

• The importance of Autoencoders,

— It finds the low dimensional representation of input data.

— Some input features may be redundant/ correlated → waste of processing time & “overfitting” in our model (too many parameters).

• If the number of Hidden UNITS is LARGE, we can impose “sparsity” constraint on hidden layer. Still the network is able to discover features.
• The network’s neurons can be active (as firing) or the neurons can be inactive.
• “sparsity parameter” usually be small value close to zero = 0.05.
• The network introduces extra penalty term to our optimization objective.
• KL(Kullback-Leibler) divergence is used in regularization, and Bernoulli random variable is considered with means from 2 distributions.
• The training should be on “whitened” natural images.
• Whitening is a “preprocessing step”, removes redundancy in the input.
• NN training strategies too work here,

1. backpropagation

2. regualarization

3. dropout

4. RBM pre-training

Other Kinds of AutoEncoders

1 ) Convolutional AutoEncoders (CAE)

• This replaces “fully-connected layer” by “convolutional layer”.
• The wide thin INOUT layer — — — converted —to — — > narrow thick hidden layer.
• This helps in extracting visual features.
• Reconstruction Quality :-

- often blurry

- lower quality

- lost of some information

• Use of CAE :-

1. Ultra-basic image reconstruction

— learns to remove noise from picture/ reconstruct missing parts.

— the input (noisy version) ; so the output (clean version).

— the network fills the gaps in the image.

2. Ultra-basic image colorization

— CAE maps circles and squares from an image to the same image but with red and blue respectively (coloring).

— Purple is formed sometimes because of blend of colors, where network hesitates between circle or square.

3. Advanced applications

— fully image colorization

— latent space clustering

— generating higher resolution images

Incase of image inputs,

— — autoencoder uses Conv layers

• Encoder = typical conv pyramid
• max-polling layer follows to reduce dimension
• decoder = deconvolutional process

— — The deconvolutional process,

- adds new data from what it had learnt previously

- is prone to visual ARTEFACTS.

- resizes using “nearest neightbour” OR “bilinear interpolation(upsampling)” , which can be better choise than standard deconvolutional layer.

2 ) Variational AutoEncoder(VAE)

• This incorporates Bayesian Inference.
• The compressed representation is a probability distribution.

3 ) Sparse AutoEncoder

• This is used for feature extraction.
• This has more hidden Units than inputs.
• This allows sparse represntation of input data.

4 ) Stacked AutoEnoder

• If more than one HIDDEN layer is used, then we seek for this Autoencoder.
• Each consecutive layer is

- optimally weighted

- non-linear

- lower dimensional projection of the training data

4 ) Deep AutoEncoders

• This has 2 symmetrical “Deep-belief networks” that has usually 4 or 5 shallow layers.
• Its layers are Restricted Boltzmann Machines (RBM).
• Uses,

1. Image search

— — An image can be compressed into around 30-number vectors (as in Google image search).

2. Data Compression

— — Deep Autoencoders are useful for “semantic hashing”.

3. Topic Modelling & Information Retrieval (IR)

References :-

1. http://ufldl.stanford.edu/tutorial/unsupervised/Autoencoders/
2. https://hackernoon.com/autoencoders-deep-learning-bits-1-11731e200694
3. https://deeplearning4j.org/deepautoencoder
4. https://lazyprogrammer.me/a-tutorial-on-autoencoders/

AutoEncoders are Essential in Deep Neural Nets was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

1 ) Deep copy & Shallow copy

• Shallow copy just creates a reference and points it towards the original object.
• But deep copy creates a separate object and points the new reference to it.

exception copy.error

• This is the exception usually raised for module specific error.

2 ) Variable length argument list

• *name should always be placed at last of the argument list.
• DOUBLE ASTERISKS ** prefixes??

• This format would be bound to a DICTIONARY, containing any keyword arguments in the actual parameters.
• Will put all the remaining last arguments with keys into a dictionary.

3 ) .pyc files (byte compiled file) OR .pyo files

• On a main python file, we import other python modules. So when we compiled/interpreted, a .pyc file for other imported modules will be created. But not the main file. This .pyc makes the loading fast on next time the main file is executed.
• This doesn’t make it run faster, but LOAD FASTER.
• CPython interpreter uses this technique.
• It caches the byte code whenever the source code has changed.
• It inserts a magic NUMBER in .pyc file. Changes whenever .pyc get changed.
• pyc files are not compatible along major Python releases.
• when there is non-matching magic number or timestamp, or the byte code version is old, .py file regenerate the .pyc files(Starts in slower process).
• To create PYC file, just import a module.

will automatically create mymodule.pyc file in the same folder

at the same time execute the module code

• byte code can be exchanged between paltforms.

>>> imp.get_magic()

will return a 4-byte string IDENTIFIER

4 ) There is no interpreted language.

• Every language can be implemented either in interpreter or compiler.

eg. There are interpreters for C, C++

There are compilers for JavaScript, PHP, Perl, Python, Ruby

• Modern languages combine them both (even multiple compilers).
• Language — a set of abstract mathematical rules.

5 ) Python module imports

• There are 3 wayes exist typically.

1. import
2. from-import
3. __import__

1) import X

— imports module X.

— creates a reference in the current namespace.

— We can call X.name to refer the the things inside the module

2) from X import *

— imports module X.

— creates a reference in the current namespace.

— We can simply use plain name to refer the things defined inside the module.

3) from X import a,b,c

— same as (2)

— can use a, b, and c.

4) X = __import__(‘X’)

— works same like import X.

— passed the module name as string.

— explicitly assigned it to a variable in current namesapce.

• Usually we can use import.
• When importing package modules

>>> from io.drivers import zip

If we run a module as a script, it is loaded under module name __main__

> give its name to interpreter

Reference :-

http://effbot.org/zone/import-confusion.htm

http://effbot.org/zone/python-compile.htm