​​As followers increased with each published work I got motivation and enthusiasm to toil harder. My genre is dynamic which incorporated several programming languages and techniques however, on medium I found a platform to demonstrate my technical skills by creating effective works.

I show my humble gratitude to all those who followed me and read my works which is undoubtedly very inspiring and appreciating. Thanks to those who followed me, thanks to the viewers too.

Steve Buyer

Large datasets are increasingly widespread in many disciplines. In order to interpret such datasets, methods are required to drastically reduce their dimensionality in an interpretable way, such as most of the information in the data is preserved. Dimensionality reduction, an unsupervised machine learning method is used to reduce the number of feature variables for each data sample selecting set of principal features.

Many techniques have been developed for this purpose, but Principal Component Analysis (PCA) is one of the oldest and most widely used. Its idea is simple to reduce the dimensionality of a dataset, while preserving as much ‘variability’ (i.e. statistical information) as possible.

1. Principal Component Analysis

PCA is a statistical technique to convert high dimensional data to low dimensional data by selecting the most important features that capture maximum information about the dataset. The features are selected on the basis of variance that they cause in the output. The feature that causes highest variance is the first principal component. The feature that is responsible for second highest variance is considered the second principal component, and so on. It is important to mention that principal components do not have any correlation with each other.

Advantages of PCA

A. Data Visualization: The challenge in today’s world is the sheer volume of data, and the variables/features that define that data. To solve a problem where data is the key, you need extensive data exploration like finding out how the variables are correlated or understanding the distribution of a few variables. Considering that there are a large number of variables or dimensions along which the data is distributed, visualization can be a challenge and almost impossible.

Hence, PCA can do that for you since it projects the data into a lower dimension, thereby allowing you to visualize the data in a 2D or 3D space with a naked eye.

B. Speeding Machine Learning (ML) Algorithm: Since PCA’s main idea is dimensionality reduction, you can leverage that to speed up your machine learning algorithm’s training and testing time considering your data has a lot of features, and the ML algorithm’s learning is too slow.

The Scikit-learn ML library provides sklearn.decomposition.PCA module that is implemented as a transformer object which learns n components in its fit() method. It can also be used on new data to project it on these components.

Performing PCA using Scikit-Learn is a two-step process:

i. Initialize the PCA class by passing the number of components to the constructor.

ii. Call the fit and then transform methods by passing the feature set to these methods. The transform method returns the specified number of principal components.

Here value of 0.99, meaning 99% of the variance of the original features has been retained. whiten=True transforms the values of each principal component so that they have 0 mean and 1 variance. If we put n_components in between 0 and 1, PCA returns the minimum amount of features that retain that much variance. Please note down we are reducing features when the data is linearly inseparable.

The PCA class contains explained_variance_ratio_ which returns the variance caused by each of the principal components. The explained_variance variable is a float type array which contains variance ratios for each principal component. The values for the explained_variance variable looks like this for first three principal components:

[0.24659629 0.18492191 0.13964843]

It can be seen that the first principal component is responsible for 24.65% variance. Similarly, the second principal component causes 18.49% variance in the dataset. Collectively we can say that 43.14% of the classification information contained in the feature set is captured by the first two principal components. Grayscale image is processed here as data.

2. Kernel PCA

Kernel Principal Component Analysis (KPCA) is a non-linear dimensionality reduction technique. It is an extension of PCA — which is a linear dimensionality reduction technique — using kernel methods. There are various kernels used in sklearn.decomposition module such as ‘linear’, polynomial kernel (‘poly’), Gaussian radial basis function kernel (‘rbf’), sigmoid kernel (‘sigmoid’), ‘cosine’, ‘precomputed’, whereas ‘linear’ is default.

The aim of PCA is to do some classification task on our data. PCA will then be useful if the data are linearly separable. Kernel PCA was developed in an effort to help with the classification of data whose decision boundaries are described by non-linear function. The idea is to go to a higher dimension space in which the decision boundary becomes linear. We have used grayscale image as data file.

3. Sparse PCA

Sparse PCA is a variant of PCA, with the goal of extracting the set of sparse components that best reconstruct the data. PCA has a disadvantage that the components extracted by PCA method have exclusively dense expressions, i.e. they have non-zero coefficients when expressed as linear combinations of the original variables. This can make interpretation difficult. In many cases, the real underlying components can be more naturally imagined as sparse vectors; for example in face recognition, components might naturally map the parts of faces. Sparse PCA is applied on make_regression dataset.

4. Incremental PCA

Incremental Principal Component Analysis (IPCA) is typically used as a replacement for PCA when the dataset to be decomposed is too large to fit in memory. IPCA builds a low-rank approximation for the input data using an amount of memory which is independent of the number of input data samples. It is still dependent on the input data features, but changing the batch size allows for control of memory usage. Incremental PCA is applied here on digits dataset.

5. Fast Independent Component Analysis

Independent Component Analysis (ICA) is used to estimate sources given noisy measurements. Imagine 3 instruments playing simultaneously and 3 microphones recording the mixed signals. ICA is used to recover the sources i.e. what is played by each instrument. Importantly, PCA fails at recovering our instruments since the related signals reflect non-Gaussian processes. Fast ICA is applied on digits dataset.

6. Linear Discriminant Analysis

Both PCA and Linear Discriminant Analysis (LDA) are linear transformation techniques: PCA is unsupervised (it ignores class labels) whereas LDA is a supervised machine learning which is used to distinguish two classes or groups.

LDA can be used to perform supervised dimensionality reduction, by projecting the input data to a linear subspace consisting of the directions which maximize the separation between classes (in a precise sense discussed in the mathematics section below). The dimension of the output is necessarily less than the number of classes, so this is in general a rather strong dimensionality reduction, and only makes sense in a multiclass setting. In scikit-learn, LDA is implemented using LinearDiscriminantAnalysis. A parameter, n_components, indicating the number of features we want returned.

7. Non-negative Matrix Factorization

Non-negative Matrix Factorization (NMF) is an alternative approach to decomposition that assumes that the data and the components are non-negative. NMF is an unsupervised technique for linear dimensionality reduction. In NMF, original data (feature matrix) breaks up into multiple matrices (i.e. factorizes) representing the latent relationship between observations and their features.

NMF can be plugged in instead of PCA or its variants, in the cases where the data matrix does not contain negative values. NMF does not provide explained variance like PCA and other techniques do, so the best way to find the optimum value of n_components is by trying a range of values. We applied NMF on breast_cancer dataset.

8. Truncated Singular Value Decomposition

Truncated Singular Value Decomposition (TSVD) is similar to PCA. This transformer performs linear dimensionality reduction by means of Truncated Singular Value Decomposition (SVD). Contrary to PCA, this estimator does not center the data before computing the singular value decomposition. This means it can work with sparse matrices efficiently. TSVD is implemented on breast_cancer dataset.

9. Performance comparison

When we have a linear question in mind, dimensionality reduction can be implemented with various PCAs techniques, LDA, and NMF. In performance comparison of various dimensionality reduction techniques, we establish linear relationship between input and output variables. We have used wine dataset. The wine dataset is a classic and very easy multi-class classification dataset. Classes are class_0, class_1, and class_2. LDA tries to identify attributes that account for the most variance between classes.

References: 1. Chris Albon, Machine Learning with Python Cookbook

Daniel Keys Moran

A data type which is expressed in numbers, rather than natural language description, is defined as Numerical data. Sometimes called quantitative data, numerical data is always collected in number form.

The sklearn.preprocessing package provides several common utility functions and transformer classes to change raw feature vectors into a representation that is more suitable for the downstream estimators.

In general, learning algorithms benefit from standardization of the data set. If some outliers are present in the set, robust scalers or transformers are more appropriate. The behavior of the different scalers, transformers, and normalizers on a data set has been observed throughout.

1. Scaling

The values of a numerical data (i.e. feature) are scaled within the range of 0 and 1.

1A. Min Max Scaler: Transform features by scaling each feature to a given range. This estimator scales and translates each feature individually such that it is in the given range on the training set, e.g. between zero and one.

Also known as min-max scaling or min-max normalization, is the simplest method and consists in rescaling the range of features to scale the range in [0, 1] or [−1, 1]. Selecting the target range depends on the nature of the data. The general formula for a min-max of [0, 1] is given as:

2. Standardizing

Features are Standardized by removing the mean and scaling to unit variance or in other words, transformation of a feature to have a mean of 0 and a standard deviation of 1.

2A. Standard Scaler: Standardization of a dataset is a common requirement for many machine learning estimators: they might behave badly if the individual features do not more or less look like standard normally distributed data (e.g. Gaussian with 0 mean and unit variance).

For instance many elements used in the objective function of a learning algorithm (such as the RBF kernel of Support Vector Machines or the L1 and L2 regularizers of linear models) assume that all features are centered around 0 and have variance in the same order. If a feature has a variance that is orders of magnitude larger that others, it might dominate the objective function and make the estimator unable to learn from other features correctly as expected.

In machine learning, we can handle various types of data, e.g. audio signals and pixel values for image data, and this data can include multiple dimensions. Feature standardization makes the values of each feature in the data have zero-mean (when subtracting the mean in the numerator) and unit-variance. This method is widely used for normalization in many machine learning algorithms (e.g., SVM, Logistic Regression, and ANN). The general method of calculation is to determine the distribution mean and standard deviation for each feature. Next we subtract the mean from each feature. Then we divide the values (mean is already subtracted) of each feature by its SD.

2B. Robust Scaler: Standardize a dataset along any axis.

3. Scaling Features to a Range

Scale each feature to the [-1, 1] range without breaking the sparsity.

This estimator scales each feature individually such that the maximal absolute value of each feature in the training set will be 1.0.

4. Normalization

The process of scaling individual samples to have unit norm is known as Normalization. This process is useful if you plan to use a quadratic form such as the dot-product or any other kernel to quantify the similarity of any pair of samples.

The function normalize provides a quick and easy way to perform this operation on a single array-like dataset, either using the l1, l2, or max norms.

5. Polynomial Features

Features created by raising existing features to an exponent are polynomial features.

6. Function Transformer (Custom Transformer)

Often, you will want to convert an existing Python function into a transformer to assist in data cleaning or processing. You can implement a transformer from an arbitrary function with FunctionTransformer.

7. Binarizer

The process of thresholding numerical features to get Boolean values is Feature Binarization. This can be useful for downstream probabilistic estimators that make assumption that the input data is distributed according to a multi-variate Bernoulli distribution.

8. Non-Linear Transformation

There are two types of Transformations: Quantile transforms and Power transforms. Both quantile and power transforms are based on monotonic transformations of the features and thus preserve the rank of the values along each feature.

8A. Quantile: Quantile Transformer provides a non-parametric transformation to map the data to a uniform distribution with values between 0 and 1.

8B. Power: In many modeling scenarios, normality of the features in a dataset is desirable. Power transforms are a family of parametric, monotonic transformations that aim to map data from any distribution to as close to a Gaussian distribution as possible in order to stabilize variance and minimize skewness.

Power transformer currently provides two such power transformations, the Yeo-Johnson transform and the Box-Cox transform.

9. Discretization

Discretization(otherwise known as quantization or binning) offers a way to partition continuous features into discrete values. Certain datasets with continuous features may benefit from discretization, because discretization can transform the dataset of continuous attributes to one with only nominal attributes.

9A. KBinsDiscretizer: This function is used to bin continuous data into intervals.

encode: Method used to encode the transformed result

1. onehot: Encode the transformed result with one-hot encoding and return a sparse matrix. Ignored features are always stacked to the right.

2. onehot-dense: Encode the transformed result with one-hot encoding and return a dense array. Ignored features are always stacked to the right.

3. ordinal: Return the bin identifier encoded as an integer value.

strategy: Strategy used to define the widths of the bins.

1. uniform: All bins in each feature have identical widths.

2. quantile: All bins in each feature have the same number of points.

3. kmeans: Values in each bin have the same nearest center of a 1D k-means cluster.

10. Label Encoder

Label Encoder is a utility class to help normalize labels such that they contain only values between 0 and n_classes-1. It can also be used to transform non-numerical labels (as long as they are hashable and comparable) to numerical labels.

11. Ordinal Encoder

We can use the Ordinal Encoder to convert categorical features to such integer codes. This estimator transforms each categorical feature to one new feature of integers.

12. One-hot Encoder

There is another possibility available to convert categorical features to features that can be used with scikit-learn estimators is to use a one-of-K, also known as one-hot or dummy encoding. This type of encoding can be obtained with the OneHotEncoder, which transforms each categorical feature with categories possible values into categories binary features, with one of them 1, and all others 0.

Jack Kilby

Classification between objects is a fairly easy task for us, but it has proved to be a complex one for machines and therefore image classification has been an important task within the field of computer vision, machine learning and image processing.

To work with images, we will go with Open Source Computer Vision Library (OpenCV). OpenCV supports a wide variety of programming languages like Python, C++, Java, etc. It can process images and videos to identify objects, faces, or even the handwriting of a human.

1. Reading Images (imread)

In this step, we simply store the path to our image dataset into a variable and then we create a function to load folders containing images into arrays so that computers can deal with it.

It is integrated with various libraries, such as NumPy which is a highly optimized library for numerical operations. Data type of the loading image (which is in grayscale) is numpy.ndarray. The image is transformed in matrix whose elements correspond to individual pixels. In grayscale images, the value of an individual elements ranges from 0 (black) to 255 (white) whereas in color images, each element contains three values corresponding to blue, green, red values (BGR).

Use the function cv2.imshow() to display an image in a window. The window automatically fits to the image size.

cv2.waitKey() is a keyboard binding function. Its argument is the time in milliseconds. The function waits for specified milliseconds for any keyboard event. If you press any key in that time, the program continues. If 0 is passed, it waits indefinitely for a key stroke.

cv2.destroyAllWindows() simply destroys all the windows we created. If you want to destroy any specific window, use the function cv2.destroyWindow() where you pass the exact window name as the argument.

2. Storing Images (imwrite)

In this step, we saves images to the filepath specified. If the filepath is same from where we are reading the image, then imwrite command will overwrite existing file without outputting an error.

3. Resizing Images (resize)

Image resizing refers to the scaling of images. Scaling comes in handy in many image processing as well as machine learning applications. It helps in reducing the number of pixels from an image and that has several advantages e.g. It can reduce the time of training of a neural network as more is the number of pixels in an image more is the number of input nodes that in turn increases the complexity of the model. Some common images sizes for ML are 32x32, 64x64, 96x96, and 256x256.

4. Cropping Images (slicing)

There is no specific function for cropping using OpenCV, NumPy array slicing is what does the job. Every image that is read in, gets stored in a 2D array (for each color channel). Simply specify the height and width (in pixels) of the area to be cropped.

5. Blurring and Sharpening Images (blur/ filter2D)

Blurring is the commonly used technique for image processing to removing the noise. It is generally used to eliminate the high-frequency content such as noise, edges in the image. The edges are being blurred when we apply blur to the image.

There are following blur functions available with OpenCV:

i.) Simple blurring (cv2.blur)

ii.) Weighted Gaussian blurring (cv2.GaussianBlur)

iii.) Median blurring (cv2.medianBlur)

iv.) Bilateral filtering (cv2.bilateralFilter)

Image Filtering is a technique to filter an image just like a one dimensional audio signal, but in 2D. In the program below, we shall learn how to filter an image using 2D Convolution with cv2.filter2D() function. The convolution happens between source image and kernel.

Kernel is another array that is usually smaller than the source image, and defines the filtering action. A kernel could be a high pass, low pass, or a custom that can detect certain features in the image. Depth is an integer variable that specifies the depth in an output image (which is -1 in this sample program).

6. Image Contrast Enhancing (equalizeHist)

OpenCV provides equalizeHist() function to easily perform Histogram Equalization of an image. It improves the contrast and brightness of an image, in order to stretch out the intensity range. This functionality helps in face detection. It flattens the graylevel histogram of an image so that all intensities are as equally common as possible. We can easily improve lightning of the low contrast image.

When we have a grayscale image, we can apply equalizeHist directly on the image but for color image we first need to convert the image to YUV (Y is luma or Brightness, U and V are color) color format. After successful conversion of the image into YUV format, we can apply equalizeHist and finally we can convert back to its original form of the image.

7. Isolating Images Colors (inRange/bitwise_and)

Isolating colors in OpenCV is simple and lucid. First, we need to convert the image to HSV (Hue, Saturation, and Value). Next, we specify some HSV values for the color blue. We create a mask, which uses an ‘inRange’ statement, for our specific range. This is true or false, black or white. Next, we ‘restore’ our blueness by running a bitwise operation. Basically, we show color where there is the frame and the mask. The white part of the mask will be blue range that was converted to pure white, while everything else became black.

8. Binarizing Images (adaptiveThreshold)

Thresholding is a technique in OpenCV, which is the assignment of pixel values in relation to the threshold value provided. In thresholding, each pixel value is compared with the threshold value. If the pixel value is smaller than the threshold, it is set to 0, otherwise, it is set to a maximum value (generally 255). Thresholding is a very popular segmentation technique, used for separating an object considered as a foreground from its background. A threshold is a value which has two regions on its either side i.e. below the threshold or above the threshold.

9. Background Removal (grabCut)

Foreground extraction refers to the process of employing image segmentation techniques and algorithms to extract the foreground (desired) and discard the background (undesired) part of an image or video feed.

Grabcut is a robust computer vision algorithm for foreground extraction. The algorithm can be summarized as below –

i. Coordinates of a rectangle enclosing the region of interest are defined (area lying outside these coordinates are automatically defined as part of the background).

ii. The parts that are classified as ‘background’ in the above step are used as a reference to classify the pixels inside the user-defined-ROI. The Gaussian mixture model is used to label pixels as probable background/foreground.

iii. Each pixel is connected to its surrounding pixels and each edge is assigned a probability of being foreground or background.

10. Edge Detection (Canny)

Edge detection is an image-processing technique, which is used to identify the boundaries (edges) of objects, or regions within an image. Edges are among the most important features associated with images. We come to know of the underlying structure of an image through its edges.

Canny Edge Detection is one of the most popular edge-detection methods in use today because it is so robust and flexible. The algorithm itself follows a three-stage process for extracting edges from an image.

i. Calculating Intensity Gradient of the Image

ii. Suppression of False Edges

iii. Hysteresis Thresholding

11. Corner Detections (cornerHarris/dilate/goodFeaturesToTrack)

OpenCV has the function cornerHarris() for the purpose of detecting corners. It takes the following parameters:

img — Input image, it should be grayscale and float32 type.

size_of_block — size of the neighbour around each pixel used for corner detection.

aperture — Aperture parameter of Sobel derivative used.

free_parameter — Harris detector free parameter in the equation (larger values correspond to identifying softer corners.)

Dilate: This operation consists of convolving an image with some kernel, which can have any shape or size, usually a square or circle.

OpenCV has a function, cv2.goodFeaturesToTrack(). It finds N strongest corners in the image by Shi-Tomasi method (or Harris Corner Detection). As usual, image should be a grayscale image. Then you specify number of corners you want to find. Then you specify the quality level, which is a value between 0–1, which denotes the minimum quality of corner below which everyone is rejected. Then we provide the minimum Euclidean distance between corners detected.

12. Feature Creation

In this part, we need to convert an image into an observation for machine learning. With the help of NumPy’s flatten function, an image containing multidimensional array convert into a vector containing the observation’s values.

If image is in grayscale, each pixel represented by one value i.e. either 1 for white or 0 for black. In case of color image, pixels are represented by three channels i.e. red, green, and blue. So in grayscale image, feature values are equal to number of pixel whereas in color image, feature values are multiple of 3 to its pixel contained.

12.1. Encoding Mean Color

Here we need to calculate the mean of red, green, and blue channel values for a color image to make three color features representing the mean color in the image. This property of encoding mean color is useful to classify images according to their colors.

12.2. Encoding Color Histograms (calcHist)

With this we can create a set of features representing the colors appearing in an image. A histogram is a representation of the distribution of values in data. It can be visualized as a plot that gives a high-level intuition of the intensity (pixel value) distribution.

OpenCV comes with an in-built cv2.calcHist() function for histogram. The parameters are:

i. img: source image of type uint8 or float32. it should be given in as a list, i.e., [loading_image].

ii. channels: it is also given in as a list []. It the index of channel for which we calculate histogram. For example, if input is grayscale image, its value is [0]. For color image, you can pass [0],[1] or [2] to calculate histogram of red, green or blue channel, respectively.

iii. mask: mask image. To find histogram of full image, it is set as None. However, if we want to get histogram of specific region of image, we should create a mask image for that and give it as mask.

iv. histSize: this represents our BIN count. Need to be given in []. For full scale, we pass [256].

v. ranges: Normally, it is [0,256].

References: 1. Chris Albon, Machine Learning with Python Cookbook

Hadley Wickham

The tf.data.Dataset API supports writing descriptive and efficient input pipelines. Dataset usage follows a common pattern:

1. Create a source dataset from your input data.
2. Apply dataset transformations to preprocess the data.
3. Iterate over the dataset and process the elements.

Iteration happens in a streaming fashion, so the full dataset does not need to fit into memory.

1. Source Datasets:

A source for datasets can be a NumPy array, tensors in memory, or some other source. If the source involves tensors, use tf.data.Dataset.from_tensors() to combine the input, otherwise use tf.data.Dataset.from_tensor_slices() if you want a separate row for each input tensor. To process lines from files, use To process lines from files, use tf.data.TextLineDataset(). On the other hand, if the input data is located on disk in a TFRecord format (which is recommended), construct a tf.data.TFRecordDataset(). If we need to create a dataset of all files matching a pattern, use tf.data.Dataset.list_files().

The tf.data.FixedLengthRecordDataset reads fixed length records from binary files and creates a dataset where each record becomes an element of the dataset. In tf.data.Dataset.from_generator, it creates a Dataset whose elements are generated by generator.

2. & 3. Transformations and Iterations:

Once you have a dataset, you can apply transformations to prepare the data for your model:

Methods

i). as_numpy_iterator() and map

A as_numpy_iterator() function returns an iterator which converts all elements of the dataset to numpy.

Map transformation applies map_func to each element of this dataset, and returns a new dataset containing the transformed elements, in the same order as they appeared in the input.

ii). filter and apply

In filter function, the dataset is filter out according to predicate. Whereas in apply function, it applies a transformation function to the dataset. A apply enables chaining of custom Dataset transformations, which are represented as functions that take one Dataset argument and return a transformed Dataset.

iii). batch

Batch function combines consecutive elements of this dataset into batches.

iv). range and take

Range function creates a Dataset of a step-separated range of values while take function creates a Dataset with at most count elements from this dataset.

v). cache

This function is used for caches the elements in this dataset.

The first time the dataset is iterated over, its elements will be cached either in the specified file or in memory. Subsequent iterations will use the cached data.

vi). flat map

It maps flat_map across the dataset and flattens the result.

vii). repeat and shuffle

In repeat, the dataset repeats so each original value is seen count times while shuffle randomly shuffles the elements of the dataset. reshuffle_each_iteration controls whether the shuffle order should be different for each epoch.

viii. Concatenate and Zip

Concatenate method creates a dataset by concatenating the given dataset with the dataset.

Zip method creates a dataset by zipping together the given datasets. This method has similar semantics to the built-in zip() function in Python, with the main difference being that the datasets argument can be a nested structure of Dataset objects.

ix. window

This function returns a dataset of “windows”. Each “window” is a dataset that contains a subset of elements of the input dataset. These are finite datasets of size size (or possibly fewer if there are not enough input elements to fill the window and drop_remainder evaluates to False).

The shift argument determines the number of input elements to shift between the start of each window. If windows and elements are both numbered starting at 0, the first element in window k will be element k * shift of the input dataset. In particular, the first element of the first window will always be the first element of the input dataset.

The stride argument determines the stride between input elements within a window.

4. TFDS

A tensorflow_datasets (tfds) defines a collection of datasets ready-to-use with TensorFlow.

Each dataset is defined as a tfds.core.DatasetBuilder, which encapsulates the logic to download the dataset and construct an input pipeline, as well as contains the dataset documentation (version, splits, number of examples, etc.).

References: 1. www.tensorflow.org

When data volume swells beyond a human’s ability to discern the patterns in it — when a company is faced with truly big data — we need a new form of intelligence. GIS, infused with artificial intelligence, can help executives make better decisions.

Mansour Raad (ESRI)

This part demonstrates the key concepts of how TensorFlow is built and how it works with complex and intuitive examples. In this part, a collection of TF 2 code samples that show you how to use various APIs that are commonly used in machine learning. More specifically, you will see a TF 2 code that involves training a CNN (Convolutional Neural Network) in the following section.

1. Mapping Function

The simplest version of map_fn repeatedly applies the callable fn to a sequence of elements from first to last. The elements are made of the tensors unpacked from elems. dtype is the data type of the return value of fn. Users must provide dtype if it is different from the data type of elems.

Comparison of speed performance of NumPy and TensorFlow is not that important part for study, as in simple calculations NumPy is little faster than TensorFlow but at the same time TensorFlow performs better in complex problems.

2. Unique and Where

Unique operation returns a tensor y containing all of the unique elements of x sorted in the same order that they occur in x; x does not need to be sorted. This operation also returns a tensor idx the same size as x that contains the index of each value of x in the unique output y.

Where operator return the elements where Condition is True (multiplexing x and y).

3. Save and Restore

TensorFlow objects may contain trackable state, such as tf.Variables, tf.keras.optimizers.Optimizer implementations, tf.data.Dataset iterators, tf.keras.Layer implementations, or tf.keras.Model implementations. These are called trackable objects.

A Checkpoint object can be constructed to save either a single or group of trackable objects to a checkpoint file. It maintains a save_counter for numbering checkpoints.

4. Tile, Concat, Stack and Repeat

Tile operation creates a new tensor by replicating input multiples times. The output tensor’s i’th dimension has input.dims(i) * multiples[i] elements, and the values of input are replicated multiples[i] times along the ‘i’th dimension. For example, tiling [a b c d] by [2] produces [a b c d a b c d].

In, tf.concat concatenates tensors along one dimension.

In Stack, it packs the list of tensors in values into a tensor with rank one higher than each tensor in values, by packing them along the axis dimension.

In tf.repeat, the command repeat elements of input.

5. Ragged Tensor

A RaggedTensor is a tensor with one or more ragged dimensions, which are dimensions whose slices may have different lengths. The total number of dimensions in a RaggedTensor is called its rank, and the number of ragged dimensions in a RaggedTensor is called its ragged-rank.

row_starts: a vector with shape [nrows], which specifies the start offset of each row. Equivalent to row_splits[:-1].

row_limits: a vector with shape [nrows], which specifies the stop offset of each row. Equivalent to row_splits[1:].

6. TFRecord and tf.train.Example

The TFRecord format is a simple format for storing a sequence of binary records. The TFRecord file format is a record-oriented binary format that you can use for training data.

Protocol buffers are a cross-platform, cross-language library for efficient serialization of structured data. Protocol messages are defined by .proto files, these are often the easiest way to understand a message type. There are two protocol buffer message types available for a TFRecord: the Example message type and the SequenceExample message type.

The tf.train.Example message (or protobuf) is a flexible message type that represents a {“string”: value} mapping. It is designed for use with TensorFlow and is used throughout the higher-level APIs such as TFX. You can store any type of data, including images, in the tf.train.Example format.

Fundamentally, a tf.train.Example is a {“string”: tf.train.Feature} mapping.

The tf.train.Feature message type can accept one of the following three types . Most other generic types can be coerced into one of these:

1. tf.train.BytesList (the following types can be coerced)

• string
• byte

2. tf.train.FloatList (the following types can be coerced)

• float (float32)
• double (float64)

3. tf.train.Int64List(the following types can be coerced)

• bool
• enum
• int32
• uint32
• int64
• uint64

7. TensorBoard: TensorFlow’s visualization toolkit

TensorBoard provides the visualization and tooling needed for machine learning experimentation:

1. Tracking and visualizing metrics such as loss and accuracy

2. Visualizing the model graph (ops and layers)

3. Viewing histograms of weights, biases, or other tensors as they change over time

4. Projecting embeddings to a lower dimensional space

5. Displaying images, text, and audio data

6. Profiling TensorFlow programs and much more

8. One Hot Encoding

Machine learning algorithms cannot work with categorical data directly. Categorical data must be converted to numbers.

A one hot encoding is a representation of categorical variables as binary vectors. This first requires that the categorical values be mapped to integer values. Then, each integer value is represented as a binary vector that is all zero values except the index of the integer, which is marked with a 1.

References: 1. www.tensorflow.org

Mark Cuban

TensorFlow is an end-to-end open source platform for Artificial Intelligence (Machine Learning) . It has a comprehensive, flexible ecosystem of tools, libraries and community resources that lets researchers push the state-of-the-art in AI and developers easily build and deploy AI powered applications.

Let us now consider the following important features of TensorFlow:

A. It includes a feature of that defines, optimizes and calculates mathematical expressions easily with the help of multi-dimensional arrays called tensors.

B. It includes a programming support of deep neural networks and machine learning techniques.

C. It includes a high scalable feature of computation with various data sets.

D. TensorFlow uses GPU computing, automating management. It also includes a unique feature of optimization of same memory and the data used.

Installing TensorFlow and Getting Started

Use pip to install “Tensorflow” in the system. The command used for installation is mentioned as below:

pip install tensorflow

Execute the following command to initialize the installation of TensorFlow (for Python version 3.5) in Anaconda:

conda create — name tensorflow python=3.5

After successful environmental setup, it is important to activate TensorFlow module.

activate tensorflow

To check the version of TensorFlow is installed, we need to use the following code:

TensorFlow 2.x (TF 2) eager execution mode makes code much easier to write compared to TF 1.x code (which used deferred execution mode).

What is Tensor?

Tensors are multi-dimensional arrays with a uniform type (i.e. the data types). It is some sorts of similarity with NumPy, tensors are (kind of) like np.arrays .

All tensors are immutable like Python numbers and strings: you can never update the contents of a tensor, only create a new one.

Scalars, Vectors, and Matrices

Scalar can be defined as one-dimensional vector, which include only magnitude and no direction. Examples weight of an object.

An array of numbers, which is either continuous or discrete, is defined as a vector. ML algorithms deal with fixed length vectors for better output generation.

Matrix can be defined as multi-dimensional arrays, which are arranged in the format of rows and columns. The size of matrix is defined by row length and column length

The rank of scalar is 0 i.e. scalar is also called as rank-0 tensor, vector is rank-1 tensor and traditionally, a rank-2 tensor is referred to as a matrix.

There is a linkage between tensors and shapes. A rank-1 tensor has a shape of dimension 1, a rank-2 tensor a shape of dimension 2, and a rank-3 tensor of dimension 3.

Data Types

The Tensorflow supports the following data types:

floating point: tf.float32, tf.float64

integers: tf.int8, tf.int16, tf.int32, tf.int64

unsigned integers: tf.uint8, tf.unit16

strings: tf.string

booleans: tf.bool

complex numbers: tf.complex64, tf.complex128

integer with quantuized ops: tf.qint8, tf.qint32, tf.quint8

1. Primitive types

TensorFlow supports tf.constant() and tf.Variable() as primitive types. tf.constant() will create tensors on the current device. Inputs which are already tensors maintain their placements unchanged. The Variable() constructor requires an initial value for the variable, which can be a tensor of any type and shape. This initial value defines the type and shape of the variable. After construction, the type and shape of the variable are fixed. The value can be changed using one of the assign methods.

TF provides a number of functions that instantiate basic tensors in memory. The simplest of these are tf.zeros() and tf.ones().

Python function const_mul() with TF 2 code that would otherwise be included in a TF 1.x tf.Session() code block. Specifically, z would be included in a sess. run() statement, along with a feed_dict that provides values for const_a and const_b. Fortunately, a decorated Python function in TF 2 makes the code look like “normal” Python code.

TF 2 eliminates global collections and their associated Application Programming Interface (API), such as tf.get_variable, tf.variable_scope, and tf.initializers. global_variables. Whenever you need a tf.Variable in TF 2, construct and initialize it directly.

2. Fill

tf.fill creates a tensor filled with a scalar value. This operation creates a tensor of shape dims and fills it with value.

3. Random Tensors Sampling

Random normal generates new sets of random values from a normal distribution. Whereas truncated normal generates sets from a truncated normal distribution.

Uniform random generates random values from a uniform distribution. The generated values follow a uniform distribution in the range (minval, maxval). The lower bound minval is included in the range, while the upper bound maxval is excluded.

In shuffle, tensor shuffles randomly along its first dimension. The tensor is shuffled along dimension 0, such that each value[j] is mapped to one and only one output[i].

4. Rank and Shape

The rank of a tensor is not the same as the rank of a matrix. The rank of a tensor is the number of indices required to uniquely select each element of the tensor. Rank is also known as “order”, “degree”, or “ndims.”

Whereas, tf.shape returns a 1-D integer tensor representing the shape of input. For a scalar input, the tensor returned has a shape of (0,) and its value is the empty vector (i.e. []).

5. Cast

It casts a tensor to a new type (i.e. data type). The operation supports data types of uint8, uint16, uint32, uint64, int8, int16, int32, int64, float16, float32, float64, complex64, complex128, bfloat16.

6. Matrix Operations

TF provides a variety of amenities for working with matrices. (Matrices by far are the most common type of tensor used in practice). In particular, TF provides shortcuts to make certain types of commonly used matrices. Various matrix operations are performed below in which matrix multiplication is important as it is widely used in ML methods.

7. Reshape

The tf.reshape does not change the order of or the total number of elements in the tensor, and so it can reuse the underlying data buffer. This makes it a fast operation independent of how big of a tensor it is operating on.

8. Imperative and Declarative

In this program, machine code instructs the machine to perform a primitive addition operation on two registers, one containing 4, and the other containing 6. The result is then 10. This style of programming is called imperative since the program tells the computer explicitly which actions to perform. In a declarative system, a computer program is a high-level description of the computation that is to be performed. It does not instruct the computer exactly how to perform the computation.

TF 2 introduced the @tf.function “decorator” for Python functions that defines a graph and performs session execution: it’s sort of a “successor” to tf.Session() in TF 1.x. Since graphs can still be useful, @tf.function transparently converts Python functions into functions that are “backed” by graphs. This decorator also converts tensor-dependent Python control flow into TF control flow, and also adds control dependencies to order read and write operations to a TF 2 state. Remember that @tf.function works best with TF 2 operations instead of NumPy operations or Python primitives.

9. Arithmetic Operations

The following program shown below is to perform the four arithmetic operations on tensors in TF 2: addition, subtraction, multiplication, and division.

Arithmetic operations involving TF 2 constants and variables can also perform.

10. Reduction Operation

TF supports different kinds of reduction. Reduction is an operation that removes one or more dimensions from a tensor by performing certain operations across those dimensions.

11. String

tf.string is a dtype, which is to say you can represent data as strings (variable-length byte arrays) in tensors. The strings are atomic and cannot be indexed the way Python strings are. The length of the string is not one of the axes of the tensor.

Please note, a ragged constant is a set of elements that have different lengths.

12. Convert to Tensor

This function converts Python objects of various types to Tensor objects. It accepts Tensor objects, numpy arrays, Python lists, and Python scalars.

13. Gradient

TF provides the tf.GradientTape API for automatic differentiation; that is, computing the gradient of a computation with respect to some inputs, usually tf.Variables. TensorFlow “records” relevant operations executed inside the context of a tf.GradientTape onto a “tape”. TF then uses that tape to compute the gradients of a “recorded” computation using reverse mode differentiation.

The variable x, followed by a with statement that initializes the variable y with the expression x*x. Next, the variable grad is initialized with the derivative that is returned by the grad_tape, and then evaluated with the current value of x.

By default GradientTape will automatically watch any trainable variables that are accessed inside the context. If you want fine grained control over which variables are watched you can disable automatic tracking by passing watch_accessed_variables=False to the tape constructor.

In case derivatives, the next code block shows you a function y, its first derivative y’, and its second derivative y’’:

y = 4*x*x*x

y’ = 4*3*x*x

y’’ = 12*2*x

14. Equal and Maximum Values

The TF 2 tf.argmax() API determines the index values containing maximum values on a row-wise basis or on a column-wise basis for a TF 2 tensor. Just to be sure you understand the previous statement: the TF 2 tf.argmax() API determines the index values that contain maximum values and not the actual maximum values in those index positions. equal() is used to perform element by equality comparison. It performs argument broadcasting before applying the comparison.

References: 1. www.tensorflow.org

Guido van Rossum

Before we begin with the code, let’s first create a vivid representation of Python Conceptual Hierarchy. Python programs can be disintegrated into modules, statements, expressions, and objects. As follows:

1. Programs are collaboration of modules.

2. Modules incorporate statements.

3. Statements comprise of expressions.

4. Expressions form and process objects.

Python’s built-in object types and some of the syntax used to code their literals — that is, the expressions that generate these objects. The following table below shows the important build-in object types:

Numbers: 12345, 1.2345, 1+2j, 0b123, Decimal(), Fraction()

Strings: ‘python’, “Mark’s”, b’a\x01c’, u’sp\xc4m’

Lists: [1, 2,[3, ‘four’], 5.6], list(range(5))

Dictionaries: {‘beverages’: ‘drink’, ‘desert’: ‘sweet’}, dict(hours=5)

Tuples: (2, ‘eggs’, 4, ‘U’), tuple(‘junk’), namedtuple

Sets: set(‘xyz’), {‘x’, ‘y’, ‘z’}

1. Immutability

Every object in Python is classified as either immutable (unchangeable) or not. In terms of the core types, numbers, strings, and tuples are immutable; lists, dictionaries, and sets are not.

2. Formatting (String Function)

The format() method formats the specified value(s) and insert them inside the string's placeholder. The placeholder is defined using curly brackets:

3. Splitting (String Function)

A split() function in Python splits the string at the specified separator, and returns a list. In this fashion, strings can be converted or split into list. Whereas rsplit() method splits a string into a list, starting from the right. If no “max” is specified, this method will return the same as the split() method. A splitlines() method splits the string at line breaks and returns a list. The splitting is done at line breaks.

4. Stripping (String Function)

Python’s strip() function is a part of built-in functions available in the Python library. The strip() method removes given characters from start and end of the original string. By default, strip() function removes white spaces from start and end of the string and returns the same string without white spaces.

5. Find and Replace ( String Function)

Python String find() is a function available in Python library to find the index of the first occurrence of a substring from the given string. The string find() function will return -1 instead of throwing an exception, if the specified substring is not present in the given string.

The Python function rfind() is similar to find() function with the only difference is that rfind() gives the highest index for the substring given and find() gives the lowest i.e the very first index.

If the value is not found, the rfind() method returns -1, but the rindex() method will raise an exception:

The Python’s replace() is an inbuilt function in the Python programming language that returns a copy of the string where all occurrences of a substring are replaced with another substring.

6. Join (on Tuple and Dictionary)

The join() method takes all items in an iterable and joins them into one string.

7. Pop (List Function)

Python list pop() is an inbuilt function in Python that removes and returns the last value from the list or the given index value.

8. Reversed

In Python Build-in function, reversed() method returns an iterator that accesses the given sequence in the reverse order.

9. Enumerate

The enumerate() method adds a counter to an iterable and returns it in a form of enumerating object. This enumerated object can then be used directly for loops or converted into a list of tuples using the list method.

10. Frozenset

The frozenset() is an inbuilt function in Python which takes an iterable object as input and makes them immutable. Simply it freezes the iterable objects and makes them unchangeable.

11. Zip and Set

The zip() function returns a zip object, which is an iterator of tuples where the first item in each passed iterator is paired together, and then the second item in each passed iterator are paired together etc.

The set() function creates a set object. The items in a set list are unordered, so it will appear in random order.

12. Bytearray and Memoryview

The bytearray() function returns a bytearray object. It can convert objects into bytearray objects, or create empty bytearray object of the specified size.

The memoryview() method returns a memory view object of the given object. The memoryview object allows Python code to access the internal data of an object that supports the buffer protocol without copying.

13. Compile

Consider a situation where we have a piece of Python code in a string and we want to compile it so that we can later run it when needed. The compile method does this task. It takes sourcecode as input and returns a code object which is ready to be executed.

14. Extend( List function)

The Python extend() method iterates over an iterable like string, list, tuple, etc., and adds each element of the iterable to the end of list.

15. Keys (Dict function)

The keys() method returns a view object. The view object contains the keys of the dictionary, as a list. The view object will reflect any changes done to the dictionary.

The fromkeys() method returns a dictionary with the specified keys and the specified value.

16. Difference (Dict function)

The difference() method returns a set that contains the difference between two sets.

The difference_update() method removes the items that exist in both sets.The difference_update() method is different from the difference() method, because the difference() method returns a new set, without the unwanted items, and the difference_update() method removes the unwanted items from the original set.

The symmetric_difference() method returns a set that contains all items from both set, but not the items that are present in both sets.

The symmetric_difference_update() method updates the original set by removing items that are present in both sets, and inserting the other items.

17. Discard and Remove (Dict function)

The built-in method, discard() in Python, removes the element from the set only if the element is present in the set (same as remove()). If the element is not present in the set, then no error or exception is raised and the original set is printed but in remove() if the element is not present in the set, then an error or exception is raised.

Yukihiro Matsumoto

Corpora are large collections of linguistic data. In fact, corpora are not always just random collections of data. Many corpora are designed to contain a careful balance of material in one or more genres.

Corpora are designed to achieve specific goal in NLP: data should provide best representation for the task. Such tasks are for example: word sense disambiguation, coreference resolution, machine translation and part of speech tagging.

1. Access to Text Corpora:

Linguistic data, or corpora contains dozens of individual texts-one per addresses. There are varieties of text corpora which we will examine one by one.

A. Gutenberg Corpus:

Project Gutenberg includes small section of texts from electronic text archive. Gutenberg contains some 25,000 free electronic books, hosted as http://www.gutenberg.org/.

First we load the NLTK package, and then file identifiers in the Gutenberg corpus:

Let’s pick up Paradise Lost by Milton and give it a short name milton, then find how many words it contains.

Book Paradise Lost contains 96825 words including symbols and punctuations.

To examine data from nltk.corpus, you have to employ the following pair of statements to perform concordance and other tasks.

Python provides another version of the import statement to avoid cumbersome to type such long names all the time.

This program displays seven statistics for each text: number of sentence, number of words used, total number of characters, number of vocabulary used, average sentence length, average word length and the number of times each vocabulary item appears in each text on an average. The raw() function gives us the contents of the file without any linguistics processing. For example, len(gutenberg.raw(‘melville-moby_dick.txt’) tells how many letters occur in the text, including the spaces between words.

The sents() function divides the text up into its sentences, where each sentence is a list of words:

B. Web and Chat Text:

NLTK’s small collection of web text includes content from a Firefox discussion forum, movie script of Pirates of the Caribbean, personal advertisements, and wine reviews:

There is a corpus of instant messaging chat sessions. The corpus contains over 10,000 posts, organized into 15 files. The filename contains the date, chatroom, and number of posts e.g. 10–19–30s_705posts.xml contains 705 posts gathered from the 30s chat room on 10/19/2006.

C. Brown Corpus:

The first million-word electronic corpus in English, created in 1961 at Brown University. This corpus contains text from 500 sources, and the sources have been categorized by genre, such as news, editorial, and so on.

The brown corpus is a meaningful resource for studying systematic differences between genres, a kind of linguistic inquiry known as stylistics.

D. Reuters Corpus:

This corpus contains 10,788 news documents totaling 1.3 million words. They are classified into 90 topics and they are grouped into two sets, called training and test.

E. Inaugural Address Corpus:

Inaugural address corpus is a collection of 55 texts, one for each presidential address.

Here we will use conditional frequency distributions to obtain counts for each corpus.

F. Annotated Text Corpus:

Many texts corpora contain linguistic annotations, representing part-of-speech tags, named entities, syntactic structures, semantic roles, and so forth. For more information and data packages containing corpora and corpus samples, download them via http://www.nltk.org/data and for how to access them, please go for Corpus HOWTO at http://www.nltk.org/howto

G. Corpus in other languages:

NLTK comes with corpora for many languages

H. Text Corpus Structure:

It is a collection of texts. Isolated structure is the simplest kind of corpus which doesn’t have any particular organization such as Gutenberg, webtext, udhr etc,. Some corpus are structured into categories such as genre (Brown corpus); some categorized as overlap, such as topic categories (Reuters Corpus), some are represented as language distributed over time (Inaugural Address corpus).

I. Creating your Own Corpus:

First load own collection of text files with the help of NLTK’s PlaintextCorpusReader. Second parameter of PlaintextCorpusReader initializer can be a list of fields, like [‘x.txt’, ‘text/x.txt’] or a pattern that matches all fields. We have to assign location of the files to corpus_root.

1.1. Conditional Frequency Distributions (CFD):

A CFD is a collection of frequency distributions, computed based on a condition. The condition will often be the category of the text such as counting words appearing in a text collection (Condition: Romance in brown corpus; cute, Monday, will appear 3, 1, 4 times respectively)|

1.2. Random Text with Bigrams:

Bigram or word pair can be created using conditional frequency distribution. The predefined bigrams() function builds a list of consecutive word pairs from a list of words available.

2. Lexical Database or Resources:

A lexicon is a collection of words and/ or phrases along with associated information, like part-of-speech and sense definitions. Lexical resources are created to enrich the text as they are secondary to texts.

Lexicon entry consists of a lemma along with additional information. Two distinct words having the same spelling are called homonyms.

e.g. Bank, where money is kept and Bank, River bank. Here Bank is Lemma

2.1. Wordlist Corpora:

NLTK includes some corpora that are nothing more than wordlists. These unusual or misspelled words are widely included in a text corpus.

There is a corpus of high frequency words (such as also, to, the) which we need to filter out before processing. This corpus contains of stopwords.

A wordlist is useful for solving word puzzles.

How many words of four letters or more can you make from these (j,a,n,m,d,s,b,p,i)? Each word must contain p.

A wordlist corpus also includes Names corpus, containing 8000 first name categorized by gender.

Try to find CFD of names corpus, where table and plot depicts the condition in which focus is on the first letter of the names.

2.2. Carnegie Mellon University (CMU) Pronouncing Dictionary:

A richer kind of lexical resource is a table or spreadsheet. NLTK includes the CMU pronouncing dictionary for U.S English. Earlier it contained 127012 words.

Observe that tunison has two pronunciations: one-syllable ‘T’, ‘AH1’, ’N’, ‘IH0’, ‘S’, ‘AH0’, ’N’, and the two-syllable ‘T’, ‘UW1’, ’N’, ‘IH0’, ‘S’, ‘AH0’, ‘N’.

The program just shows the scans of lexicon looking for dict_entries whose pronunciation consists of five phones (phonetic codes of distinct labels).

The phones contain digits to represent primary stress as 1, secondary stress as 2, and no stress as 0.

2.3. Comparative Wordlists:
Comparative wordlist is another example of a tabular lexicon. NLTK includes lists of about 200 common words in several languages or we called it as Swadesh wordlists.

We can convert languages into a simple dictionary.

We can add other source languages to translator.

Here we can compare words in various languages in particular sequence.

2.4. Toolbox Lexicons:
A toolbox file consists of a collection of entries, where each entry is made up of one or more fields. Most fields are repeatable or optional, which means this lexical resources cannot be treated as a table or spreadsheet.

The Rotokas language is spoken on the island of Bouganville, Papua New Guinea.

3. WordNet:

NLTK includes the English WordNet, with 155,287 words and 117,659 synonym sets. WordNet is a semantically oriented dictionary of English, similar to a traditional thesaurus but with richer structure.

3.1. Sense and Synonyms:
We replace a word cloth in sentence (A.) with fabric and material to get (B.) and (C.) respectively. The meaning of the sentence remains pretty much same.

A. The cloth of a dress is soft and adorable
B. The fabric of a dress is soft and adorable
C. The material of a dress is soft and adorable

We can explore these words using WordNet

The entity fabric.n.01 and framework.n.02 are called a synset i.e. synonym set, a collection of synonymous words or lemmas.

WordNet navigates the lexical relations between the synset such as hyponyms, hypernyms, meronyms, holonyms, entails and antonyms.
Hyponyms give abstract concepts of the word that are more specific.

Some words have multiple paths, because they can be classified in more than one way. Hypernyms and hyponyms are called lexical relations because they relate one synset to another.

Another way to navigate the WordNet network is from items to their components i.e. meronyms and to the things they are contained in i.e. holonyms.

There are also relationships between verbs. For example, the act of writing involves the act of spelling, so writing entails spellings.

Some lexical relationships hold between lemmas e.g. antonyms

4. Semantic Similarity:

Semantic Similarity, or Semantic Textual Similarity, is a task in the area of NLP that scores the relationship between texts or documents using a defined metric.
In the following example panther is very specific, whereas feline is more general and physical_entity is completely general.

We can quantify this concept of generality by looking up the depth of each synset.

Similarity measures have been defined over the collection of WordNet synsets that incorporate this insight. path_similiarty assigns score in the range 0–1 based on the shortest path that connects the concepts in the hypernym hierarchy.

References: 1. Steven bird, Ewan klein, and Edward loper; Natural Language Processing with Python

Natural Language Processing (NLP) is a branch of artificial intelligence that helps computers understand, interpret and manipulate human language. NLP draws from many disciplines, including computer science and computational linguistics, in its pursuit to fill the gap between human communication and computer understanding.

Why is NLP important?

Large volumes of textual data

NLP helps computers communicate with humans in their own language and scales other language-related tasks. For example, NLP makes it possible for computers to read text, hear speech, interpret it, measure sentiment and determine which parts are important.

Structuring a highly unstructured data source

Human language is astoundingly complex and diverse. We express ourselves in infinite ways, both verbally and in writing. There are not only hundreds of languages and dialects, but within each language there is a unique set of grammar and syntax rules, terms and slang. When we write, we often misspell or abbreviate words, or omit punctuation. When we speak, we have regional accents, and we mumble, stutter and borrow terms from other languages.

How does NLP work?

These underlying tasks are often used in higher-level NLP capabilities, such as:

· Content categorization — A linguistic-based document summary, including search and indexing, content alerts and duplication detection.

· Topic discovery and modelling — Accurately captures the meaning and themes in text collections, and apply advanced analytics to text, like optimization and forecasting.

· Contextual extraction — Automatically pulls structured information from text-based sources.

· Sentiment analysis — Identifying the mood or subjective opinions within large amount of text, including average sentiment and opinion mining.

· Speech-to-text and text-to-speech conversion — Transforming voice commands into written text, and vice versa.

· Document summarization — Automatically generating synopses of large bodies of text.

· Machine translation — Automatic translation of text or speech from one language to another.

Getting started with Natural Language Toolkit (NLTK)

NLTK is a leading platform for building Python programs to work with human language data.

First you have to install nltk, for this use following command

pip install nltk

For anaconda user, it is already installed so you don’t need to bother otherwise try the following command mentioned below

conda install nltk

once you have done with installation part, import nltk library in Python interpreter and download the required data for book (in our case)

Note: Here we will treat text as raw data for the program

One the data is downloaded to your machine, use the following command, it will load all items from NLTK’s book module

We want to find out about these texts

Searching Text

There are various ways to examine the content of a text. A concordance view shows every occurrence of a given word, together with some context.

Distributional similarity: find other words which appear in the same contexts as the specified word.

In common contexts, command find context where all the specified words can appear; and return a frequency distribution mapping each context to the number of times the context was used. Whereas, dispersion plot is a plot showing the distribution of the words through the text.

Let’s try to generate some random text in various styles. Once text is generated it will appear same for number of times you would run this command.

Counting Vocabulary

In this section, we will play with different texts of the book module which are differing in the vocabulary they use. After this you have a new sense of the richness and diversity of language.

The term len provides the length of something, which we will apply here to the book of Chat corpus.

So Chat corpus has 45010 words and punctuation symbols, or “token”. A token is the technical name for a sequence of characters- such as beautiful, her, or +.

With sorted command, text is sorted in the list of vocabulary items, start with punctuation symbols and thereafter with A and so on.

This shows that each word is used around 7 times on average

The block of code that does a task (like calculating lexical diversity or percentage in our case) is called function.

A Closer Look at Python: Texts as Lists of Words

Lists

Indexing Lists

Now, use an index that is too large, we get an error

A closer look at slicing, the slice 3:7 includes sentence elements at indexes 3, 4, 5, 6 and 7

We can modify the element of a list by assigning to one of its index values. We can replace the entire slice with new material.

The consequence of this last change is that the list has only four elements.

Variable

Variable can be equal to expression. The name of variable can be anything you like e.g. sentence, Another_SentencE

Remember that capitalized words appear before lowercase words in sorted list.

Strings

We used to access the elements of a listed work with individual words, or strings. We can assign a string to a variable, index a string, slice a string, or perform multiplication and addition with strings and we can join the words of a list to make a string and split the string into a list.

Computing with language: Statistics

In nltk, FreqDist provides you with the raw counts (i.e. frequencies of words) in the text. It is a ‘distribution’ since it tells us how the total number of word tokens in the text are distributed across the vocabulary items.

Hapaxes

Hapaxes are the words occur once only. We can see that there are 3408 words out of 69213 in text9 which occur once only as shown below in the program.

Fine-Grained Selection of Words

We would like to find the words from the vocabulary of the text which are more than let’s say 14 characters long.

Let’s call this property P, so that P (w) is true if and only if w is more than 14 characters long.

w for w in V if P(w)

This means “the set of all w such that w is an element of Vocab (the vocabulary) and w has property P.”

In this example, len(word)>9 ensures that the words are longer than nine words and Freq_Dist_1[word] ensures that these words occur more than ten times.

Collocations and Bigrams

A collocation is a sequence of words that occur together unusually often. Thus tube light is a collocation. To get a handle on collocations, we start off by extracting from a text a list of word pairs, also known as bigrams.

Counting other things

Making Decisions and Taking Control

Conditional

Comparison operator

Nested Code Blocks

Mostly programming languages execute a block of code when a conditional expression, or if statement, is satisfied.

Looping with Conditions

References: 1. Steven bird, Ewan klein, and Edward loper; Natural Language Processing with Python