Tensors in Deep Learning – Marko Kovacevic Blog

1. What is Tensor

Tensors are primary data structures used by Artificial Neural Networks. All inputs and outputs are represented using Tensors.

Generally Tensors are multi dimensional arrays.

2. Difference between Tensor and other data structures

*Difference between Scalar, Vector, Matrix and Tensor*

The table below compare same data structures but different names comparing to Computer science and Mathematics. Also in table is displayed how much Indexes are required to access specific element of data structure.

Indexes required	Computer science	Mathematics
0	number	scalar
1	array	vector
2	2d-array	matrix
n	nd-array	nd-tensor

Same data structures but different names

When more than two indexes are required to access a specific element, we stop giving specific names to the structures, and we begin using more general language.

The reason we say a Tensor is a generalization, because we use the word Tensor for all values of n like so:

Scalar is a 0 dimensional Tensor
Vector is a 1 dimensional Tensor
Matrix is a 2 dimensional Tensor
nd-array is an n dimensional Tensor

In Deep learning, Tensors and nd-arrays are the same thing!

Tensors allow us to drop these specific terms and just use an n to identify the number of dimensions we are working with.

Dimension of a Tensor differs from what we mean when we refer to the dimension of a vector in a vector space. In next section will be explained how data is represented with Tensors.

3. How data is represented with Tensors

Tensors are good because everything can be represented as a Tensor.

Image below shows how Tensors with different dimensions looks.

The table below shows how Tensors are repsresented in Python.

Rank	Math entity	Python example
0	Scalar (magnitude only)	s = 483
1	Vector (magnitude and direction)	v = [1.1, 2.2, 3.3]
2	Matrix (table of numbers)	m = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
3	3-Tensor (cube of numbers)	t = [[[2], [4], [6]], [[8], [10], [12]], [[14], [16], [18]]]
n	n-Tensor (you get the idea)	….

How Tensors are represented in Python

For example text, images, videos can be represented as a Tensor.

3.1. Rank

Rank of a Tensor is the number of dimensions that Tensor have. Suppose we have a Tensor Rank of 2. This means all of the following:

We have a matrix
We have a 2d-array
We have a 2d-tensor

Tensor Rank tells us how many indexes are needed to refer to a specific element within the Tensor.

3.2. Axes

Axis of a Tensor is a specific dimension of a Tensor.

If we have a Tensor and we want to refer to a specific dimension, we use the word Axis in Deep learning.

If Tensor has Rank 2 then it has 2 Axes .

If first Axis has length of 3 then we can index three positions:

t[0]
t[1]
t[2]

If second Axis has length of 2 then we can index two positions in the second axis:

t[0][0]
t[1][0]
t[2][0]

t[0][1]
t[1][1]
t[2][1]

3.3. Shape

Shape of a Tensor gives us the length of each Axis of Tensor.

If we have Tensor:

dd = [
  [1,2,3],
  [4,5,6],
  [7,8,9]
]

Then it has Shape 3×3.

3.4. DType

DType of Tensor represent type of elements in a Tensor.

In TensorFlow framework DType objects are defined:

tf.float16: 16-bit half-precision floating-point.
tf.float32: 32-bit single-precision floating-point.
tf.float64: 64-bit double-precision floating-point.
tf.bfloat16: 16-bit truncated floating-point.
tf.complex64: 64-bit single-precision complex.
tf.complex128: 128-bit double-precision complex.
tf.int8: 8-bit signed integer.
tf.uint8: 8-bit unsigned integer.
tf.uint16: 16-bit unsigned integer.
tf.uint32: 32-bit unsigned integer.
tf.uint64: 64-bit unsigned integer.
tf.int16: 16-bit signed integer.
tf.int32: 32-bit signed integer.
tf.int64: 64-bit signed integer.
tf.bool: Boolean.
tf.string: String.
tf.qint8: Quantized 8-bit signed integer.
tf.quint8: Quantized 8-bit unsigned integer.
tf.qint16: Quantized 16-bit signed integer.
tf.quint16: Quantized 16-bit unsigned integer.
tf.qint32: Quantized 32-bit signed integer.
tf.resource: Handle to a mutable resource.
tf.variant: Values of arbitrary types.

3.5. Real world examples of Tensors

3.5.1. Scalars (0D Tensor)

Scalars are Tensors that do not have dimension, they are only a number.

For example current year 2020.

3.5.1. Vectors (1D Tensor)

Vectors are represented in 1D Tensor and they have shape of (samples).

If we use example of 7 lottery numbers [10, 14, 6, 12, 26, 3, 9] then shape will be (7).

3.5.2. Matrices (2D Tensor)

Matrices are represented in 2D Tensor and they have shape of (samples, features).

If we use example of dataset about persons and we store 3 informations about each person (age, ZIP code and income) and entire dataset has 1000 persons then shape of Tensor would be (100000, 3).

3.5.3. Time-series (3D Tensor)

Time-series are represented in 3D Tensor and they have a shape of (samples, timesteps, features).

If we use example of stock prices then for every day and every minute we will store the current price of the stock, the highest price in the past minute, and the lowest price in the past minute. That will be stored in a 3D Tensor of shape (250,390, 3) where 250 is days, 390 is minutes for one day and 3 is number of features that is stored.

3.5.4. Image (4D Tensor)

Images are represented in 4D Tensor and they have a shape of (samples, height, width, channels).

Images typically have 3 dimensions: height, width, and colour depth. Grayscale Images contains only one channel. RGB Images has three colour channels.

A batch of 128 grayscale images of size 256 × 256 could be stored in a Tensor of shape (128, 256, 256, 1) and a batch of 128 color images could be stored in a Tensor of shape (128, 256, 256, 3)

RGB Images : (128, 256, 256, 3)
Grayscale Images : (128, 256, 256, 1)

3.5.5. Video (5D Tensor)

Videos are represented in 5D Tensor and they have a shape of (samples, frames, height, width, color_depth).

For videos is needed 5D Tensors. Video can be understood as a sequence of frames, each frame is color image that can be stored in a 3D Tensor (height, width, color_depth), sequence of frames can be stored in a 4D Tensor (frames, height, width, color_depth) and a batch of different videos can be stored in a 5D Tensor of shape (samples, frames, height, width, color_depth).

YouTube logo

YouTube video of 144 × 256 resolution which lasts 60 seconds and have 4 frames per second would have 240 (60*4) frames. Batch of 5 videos would be stored in a Tensor of shape (5, 240, 144, 256, 3). That would have total 132710400 (5*240*144*256*3) values. If the dtype of the Tensor was float32, then each value would be stored in 32 bits which is 4246732800 bits (132710400*32), so the Tensor would represent 506,25 MB (4246732800 /8 /1024 *1024). That is heavy and also that is one of couple challenges that Artificial intelligence engineers have when building software based on Deep learning. But videos in real life are much lighter because they are not stored in float32, they re typically compressed by MPEG format.

4. How Tensors are efficient in computations

Tensors are efficient in computations because Linear algebra can be very fast on GPUs.

5. How TensorFlow got name

TensorFlow, popular framework for Deep learning, was named from Tensor and Flow. Where Flow mean that Tensor (data) flow through the graph of operations.

Thanks for reading this post.

6. References

Code · Data Science. 2020. Deep Learning Book Series · 2.1 Scalars Vectors Matrices And Tensors. [online] Available at: <https://hadrienj.github.io/posts/Deep-Learning-Book-Series-2.1-Scalars-Vectors-Matrices-and-Tensors/> [Accessed 3 May 2020].
DataCamp Community. 2020. Investigating Tensors With Pytorch. [online] Available at: <https://www.datacamp.com/community/tutorials/investigating-tensors-pytorch> [Accessed 4 May 2020].
Deeplizard.com. 2020. Tensors Explained – Data Structures Of Deep Learning. [online] Available at: <https://deeplizard.com/learn/video/Csa5R12jYRg> [Accessed 4 May 2020].
Deeplizard.com. 2020. Rank, Axes, And Shape Explained – Tensors For Deep Learning. [online] Available at: <https://deeplizard.com/learn/video/AiyK0idr4uM> [Accessed 4 May 2020].
mc.ai. 2020. Tensors — Representation Of Data In Neural Networks. [online] Available at: <https://mc.ai/tensors%E2%80%8A-%E2%80%8Arepresentation-of-data-in-neural-networks/> [Accessed 5 May 2020].
Chromium.googlesource.com. 2020. Tensorflow – Tensor Ranks, Shapes, And Types. [online] Available at: <https://chromium.googlesource.com/external/github.com/tensorflow/tensorflow/+/r0.7/tensorflow/g3doc/resources/dims_types.md> [Accessed 5 May 2020].
TensorFlow. 2020. Tf.Dtypes.Dtype | Tensorflow Core V2.1.0. [online] Available at: <https://www.tensorflow.org/api_docs/python/tf/dtypes/DType> [Accessed 6 May 2020].