ANNs, CNNs, RNNs

Neural Networks

Different types of neural networks, for specific tasks:

• Artificial Neural Networks (ANNs) – general purpose
• Convolutional Neural Networks (CNNs) – spatial, image data
• Recurrent Neural Networks (RNNs) – sequential data (text, audio, time series)

Node (or Neuron)

A node is like a simple decision-making unit that:

• takes multiple inputs
• assigns importance to those inputs through weights
• decides whether to “activate” or not based on those weighted inputs

each input gets a value proportional to its weight, and if the total values cross a certain threshold the node “fires” or produces an output

ANNs layers

• Dense Layers form the core of most neural networks, each node is connected to every node in the previous layer
• Activation Layers activates non-linear transformations, can be used separately or within dense layers
• Dropout and Normalization Layers help improve model performance and generalization by for example randomly setting input units to 0 during training – this prevents overfitting, makes models more generalizable

ANNs layers in tensorflow

Dense layer:

tf.keras.layers.Dense(units=10)

We can add an activation to a Dense layer:

tf.keras.layers.Dense(128, activation="relu")

ANNs layers in tensorflow

Stand alone activation layers:

tf.keras.layers.ReLU()
tf.keras.layers.Sigmoid()
tf.keras.layers.Tanh()
tf.keras.layers.LeakyReLU(alpha=0.1)

ANNs layers in tensorflow

Dropout layer:

# randomly drops 50% of inputs during training
tf.keras.layers.Dropout(rate=0.5)  

Normalization layer (to stabilize and speed up training):

# normalizes the inputs of each layer
tf.keras.layers.LayerNormalization()

ANNs layers in tensorflow

Regularization Layers (L1 and L2), control model complexity.

L1 (Lasso) regularization – adds the absolute value of the coefficients as penalty, promoting sparsity. It sets some coefficients to zero, which results in feature selection.

tf.keras.layers.Dense(units=64, 
    kernel_regularizer=tf.keras.regularizers.l1(0.01)
)

L2 (Ridge) regularization – adds the squared value of the coefficients as penalty. It shrinks coefficients towards zero.

tf.keras.layers.Dense(units=64, 
    kernel_regularizer=tf.keras.regularizers.l2(0.001)
)

CNNs layers

In addition to normalization layers (to stabilize training) and dropout layers (to prevent overfitting), here are the layers we use in CNNs:

• Convolutional Layers – to extract spacial features
• Pooling – to reduce spatial dimensions

CNNs layers in tensorflow

Convolutional Layers, 1D:

tf.keras.layers.Conv1D(filters=64, kernel_size=3, activation='relu')

Convolutional Layers, 2D:

tf.keras.layers.Conv2D(filters=64, kernel_size=(3, 3), activation='relu')

Convolutional Layers, 3D:

tf.keras.layers.Conv3D(filters=64, kernel_size=(3, 3, 3), activation='relu')

CNNs layers in tensorflow

Pooling (max):

tf.keras.layers.MaxPooling2D()

Case Study – Flower image classification

• Download the images and set them up in a project folder
• Create a new python script file
• Import tensorflow

import tensorflow as tf

Reading images in

Keras has a handy image_dataset_from_directory() method. We have to speficy batch size, image height and width (the method will resize all images for us):

img_height = 180
img_width = 180

train_ds = tf.keras.utils.image_dataset_from_directory(
      "flower_photos",
      validation_split=0.2,
      subset="training",
      seed=123,
      image_size=(img_height, img_width),
      batch_size=32
  )

We can get the class names from the data:

class_names = train_ds.class_names
print(class_names)

Model creation

Let’s rescale the pixel values, and include Conv2D and tf.keras.layers.MaxPooling2D layers to process the 2D data. Then we include a Dropout layer. Flatten and Dense are layers we have used for the digits problem. Then our final Dense has 5 nodes because we have 5 classes.

model = tf.keras.Sequential([
        tf.keras.layers.Rescaling(1./255, input_shape=(img_height, img_width, 3)),
        tf.keras.layers.Conv2D(16, 3, padding='same', activation='relu'),
        tf.keras.layers.MaxPooling2D(),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dense(128, activation='relu'),
        tf.keras.layers.Dense(num_classes)
    ])

Compile model

We will compile our model like you did before:

model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

And then fit our training data:

model.fit(train_ds, epochs=10)

Test the model

Here’s the code to test the model with a new image:

img = tf.keras.utils.load_img("test_flowers/tulips.jpeg", 
                               target_size=(img_height, img_width))

img_array = tf.keras.utils.img_to_array(img)
img_array = tf.expand_dims(img_array, 0) # create a batch

predictions = model.predict(img_array)
score = tf.nn.softmax(predictions[0])

print(
  "This image most likely belongs to {} with a {:.2f} percent confidence."
  .format(class_names[np.argmax(score)], 100 * np.max(score))
  )