Skip to content

Temporal layers

This section describes the functionality of temporal graph layers provided by the framework.

General information#

Temporal layers are based and adapted from the paper of Spatio-Temporal Graph Convolutional Networks and extended to work with different temporal and graph layers. Generally, each spatio-temporal-block consists of 3 layers as seen here.

Architecture of spatio-temporal block

Temporal layers can be either of the following TF/Keras layers:

Graph layers can be either of:

Spatio-temporal blocks can be stacked and work for graphs that only contain node features, as well as graphs with both node and edge features.

Convolutional Spatio-Temporal Blocks#

Layers that extend the TemporalConv layer implement the spatio-temporal block by using 2D-Convolutions as temporal layers.

The first convolution layer receives a sequence of graphs in the dimension (batch_size, seq_len, n_nodes, embedding_size) and outputs the same dimension again. For that, it uses seq_len as the filter count, and a 2D kernel of size (1, embedding_size), meaning that the node features of each graph are handled sequentially independent of each other. To keep the input dimension, "same"-padding is applied.

The graph layer receives all graphs in the learned sequence and stacks them to transform their node_features. The output dimensions is (batch_size * seq_len, n_nodes, embedding_size).

The second temporal layer works identically to the first, except that the number of filters is determined by the hyperparameter output_seq_len which determines how many graphs are in the output sequence.

Recurrent Spatio-Temporal Blocks#

Layers that extend GraphLSTM, GraphGRU or GraphConvLSTM work slightly differently.

The first recurrent layer operates node-wise, meaning that for each graph in the sequence, it receives node features of the same node until all nodes are processed. For a sequence of 3 graphs, denoted by node#node (#graph), it would process nodeA(0), nodeA(1), nodeA(2), nodeB(0), etc. After processing the whole sequence of the layer is returned.

This sequence is reshaped back into (batch_size * seq_len, n_nodes, n_node_features) for the node features to be transformed by the graph layer. The output is again (batch_size * seq_len, n_nodes, embedding_size).

The second recurrent layer then works the same as the first. Since the used recurrent layers are limited to single-step predictions out-of-the-box, the RecurrentOutputBlock is required at the model output to produce an output sequence containing multiple graphs.

NOTE: The mentioned dimensionalities do not apply to the GraphConvLSTM since it uses a 5D input instead of 3D. It receives an additional empty channel dimension. Its convolution is configured to work on a single filter only, but using the same dimensions as described for the Spatio-Temporal Convolution Blocks.

Output Blocks#

Since the procedure of spatio-temporal blocks is unique, the framework provides two output layers:

For their exact functionality, check the respective documentations.

Edge features in spatio-temporal blocks#

As shown in the tutorial on model definition, edge features are fed into each layer manually. They are used to create a weighted adjacency matrix which in turn strengthens or weakens the importance of some nodes of a graph. This is also true for the Spatio-Temporal Blocks, however there is an additional step because of the parameter output_seq_len. Since edge features are not part of the output of each block, their sequence length is constant. Therefore, when stacking layers they are passed through a convolutional layer to ensure that the sequence length for the edge features is equal to the sequence length of the node features.