I’m looking through some different neural network architectures and trying to piece together how to recreate them on my own.
One issue I’m running into is the functional difference between the Concatenate() and Add() layers in Keras. It seems like they accomplish similar things (combining multiple layers together), but I don’t quite see the real difference between the two.
Here’s a sample keras model that takes two separate inputs and then combines them:
inputs1 = Input(shape = (32, 32, 3))
inputs2 = Input(shape = (32, 32, 3))
x1 = Conv2D(kernel_size = 24, strides = 1, filters = 64, padding = "same")(inputs1)
x1 = BatchNormalization()(x1)
x1 = ReLU()(x1)
x1 = Conv2D(kernel_size = 24, strides = 1, filters = 64, padding = "same")(x1)
x2 = Conv2D(kernel_size = 24, strides = 1, filters = 64, padding = "same")(inputs2)
x2 = BatchNormalization()(x2)
x2 = ReLU()(x2)
x2 = Conv2D(kernel_size = 24, strides = 1, filters = 64, padding = "same")(x2)
add = Concatenate()([x1, x2])
out = Flatten()(add)
out = Dense(24, activation = 'softmax')(out)
out = Dense(10, activation = 'softmax')(out)
out = Flatten()(out)
mod = Model([inputs1, inputs2], out)
I can substitute out the Add() layer with the Concatenate() layer and everything works fine, and the models seem similar, but I have a hard time understanding the difference.
For reference, here’s the plot of each one with keras’s plot_model function:
KERAS MODEL WITH ADDED LAYERS:
KERAS MODEL WITH CONCATENATED LAYERS:
I notice when you concatenate your model size is bigger vs adding layers. Is it the case with a concatenation you just stack the weights from the previous layers on top of each other and with an Add() you add together the values?
It seems like it should be more complicated, but I’m not sure.
>Solution :
As you said, both of them combine input, but they combine in a different way.
their name already suggest their usage
Add() inputs are added together,
For example (assume batch_size=1)
x1 = [[0, 1, 2]]
x2 = [[3, 4, 5]]
x = Add()([x1, x2])
then x should be [[3, 5, 7]], where each element is added
notice that the input shape is (1, 3) and (1, 3), the output is also (1, 3)
Concatenate() concatenates the output,
For example (assume batch_size=1)
x1 = [[0, 1, 2]]
x2 = [[3, 4, 5]]
x = Concatenate()([x1, x2])
then x should be [[0, 1, 2, 3, 4, 5]], where the inputs are horizontally stacked together,
notice that the input shape is (1, 3) and (1, 3), the output is also (1, 6),
even when the tensor has more dimensions, similar behaviors still apply.
Concatenate creates a bigger model for an obvious reason, the output size is simply the size of all inputs summed, while add has the same size with one of the inputs
For more information about add/concatenate, and other ways to combine multiple inputs, see this