I am trying to train a model with keras and the data I am feeding it is of the following type. I hae a dataframe with input an input vector called Fingerprint and an ouput vector called position. For referenceFingerprint are recieved signal intensities from individual devices at given positions, position.
index deviceid \
0 0 A4060463730116275
1 85 A4050490185016277
2 185 A4060463594816270
3 213 A4050470924716273
4 263 A4060455554316272
5 300 A4050500829316277
6 384 A4050500938216277
7 451 A4610490003016278
8 500 A4050494117516271
9 602 A4060463727316273
Fingerprint \
0 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.73, 0.0, 2.18, 0.0, 0.0, 0.0, 2.47, 11.09, 15.56, 5.4, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
1 [0.0, 0.0, 0.0, 2.25, 0.0, 5.0, 8.0, 8.53, 15.0, 9.0, 0.0, 1.91, 0.0, 10.29, 3.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
2 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 15.53, 15.77, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
3 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 7.0, 12.0, 2.4, 4.47, 0.0, 0.0, 0.0, 0.0]
4 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 15.53, 14.0, 4.17, 0.0, 0.0, 0.0, 0.0, 0.0]
5 [10.0, 14.0, 15.44, 7.6, 2.27, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0, 11.09]
6 [0.0, 10.0, 11.0, 15.6, 6.23, 2.11, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 4.0]
7 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 0.0, 10.0, 15.5, 7.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
8 [0.0, 0.0, 0.0, 0.0, 0.0, 3.67, 15.47, 0.0, 15.53, 15.53, 0.0, 14.0, 0.0, 9.0, 5.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
9 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.75, 0.0, 6.0, 15.62, 0.0, 9.0, 0.0, 3.53, 12.38, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
manual_x manual_y position
0 -16.8169 16.2185 [-16.8169, 16.2185]
1 1.9891 30.7416 [1.9891, 30.7416]
2 10.1897 -2.2842 [10.1897, -2.2842]
3 19.1583 -6.5890 [19.1583, -6.589]
4 4.3096 0.8620 [4.3096, 0.862]
5 16.2592 37.6467 [16.2592, 37.6467]
6 18.5052 40.0921 [18.5052, 40.0921]
7 -15.7257 18.1974 [-15.7257, 18.1974]
8 3.9464 21.6294 [3.9464, 21.6294]
9 9.2405 9.3424 [9.2405, 9.3424]
​
(as a dict :
{'index': {0: 0, 1: 85, 2: 185, 3: 213, 4: 263, 5: 300, 6: 384, 7: 451, 8: 500, 9: 602}, 'deviceid': {0: 'A4060463730116275', 1: 'A4050490185016277', 2: 'A4060463594816270', 3: 'A4050470924716273', 4: 'A4060455554316272', 5: 'A4050500829316277', 6: 'A4050500938216277', 7: 'A4610490003016278', 8: 'A4050494117516271', 9: 'A4060463727316273'}, 'Fingerprint': {0: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.73, 0.0, 2.18, 0.0, 0.0, 0.0, 2.47, 11.09, 15.56, 5.4, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 1: [0.0, 0.0, 0.0, 2.25, 0.0, 5.0, 8.0, 8.53, 15.0, 9.0, 0.0, 1.91, 0.0, 10.29, 3.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 2: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 15.53, 15.77, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 3: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 7.0, 12.0, 2.4, 4.47, 0.0, 0.0, 0.0, 0.0], 4: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 15.53, 14.0, 4.17, 0.0, 0.0, 0.0, 0.0, 0.0], 5: [10.0, 14.0, 15.44, 7.6, 2.27, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0, 11.09], 6: [0.0, 10.0, 11.0, 15.6, 6.23, 2.11, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 4.0], 7: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 0.0, 10.0, 15.5, 7.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 8: [0.0, 0.0, 0.0, 0.0, 0.0, 3.67, 15.47, 0.0, 15.53, 15.53, 0.0, 14.0, 0.0, 9.0, 5.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 9: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.75, 0.0, 6.0, 15.62, 0.0, 9.0, 0.0, 3.53, 12.38, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}, 'manual_x': {0: -16.8169, 1: 1.9891, 2: 10.1897, 3: 19.1583, 4: 4.3096, 5: 16.2592, 6: 18.5052, 7: -15.7257, 8: 3.9464, 9: 9.2405}, 'manual_y': {0: 16.2185, 1: 30.7416, 2: -2.2842, 3: -6.589, 4: 0.862, 5: 37.6467, 6: 40.0921, 7: 18.1974, 8: 21.6294, 9: 9.3424}, 'position': {0: [-16.8169, 16.2185], 1: [1.9891, 30.7416], 2: [10.1897, -2.2842], 3: [19.1583, -6.589], 4: [4.3096, 0.862], 5: [16.2592, 37.6467], 6: [18.5052, 40.0921], 7: [-15.7257, 18.1974], 8: [3.9464, 21.6294], 9: [9.2405, 9.3424]}}
So, to train my model I split my data in the following way:
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(Ack_record_with_manual.iloc[:,2:3].values, Ack_record_with_manual.iloc[:,5:6].values,test_size=0.2)
For instance, X_train is given by:
array([[list([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 2.0, 7.0, 3.0, 0.0, 0.0, 0.0, 14.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])],
[list([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 13.0, 15.58, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])],
[list([2.0, 0.0, 0.0, 0.0, 14.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.0, 15.0, 14.0])],
...,
[list([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 6.0, 15.67, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])],
[list([0.0, 0.0, 0.0, 0.0, 12.0, 15.91, 5.31, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 7.91, 0.0, 5.17])],
[list([5.57, 5.38, 2.0, 2.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 12.33, 13.0])]],
dtype=object)
But, this is an issue since I cannot pass this to the following model:
model = Sequential()
model.add(Dense(20, input_dim=1, activation='relu', kernel_initializer='he_uniform'))
model.add(Dense(1, activation='linear'))
opt = SGD(lr=0.01, momentum=0.9)
model.compile(loss='mean_squared_error', optimizer=opt)
# fit model
history = model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=100, verbose=0)
# evaluate the model
train_mse = model.evaluate(X_train, y_train, verbose=0)
test_mse = model.evaluate(X_test, y_test, verbose=0)
print('Train: %.3f, Test: %.3f' % (train_mse, test_mse))
# plot loss during training
pyplot.title('Loss / Mean Squared Error')
pyplot.plot(history.history['loss'], label='train')
pyplot.plot(history.history['val_loss'], label='test')
pyplot.legend()
pyplot.show()
because it returns the following error:
ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type list).
Hence, I tried to convert X_train to a tensor by doing the following:
import numpy as np
def my_func(arg):
arg = tf.convert_to_tensor(arg, dtype=tf.float32)
return arg
X_train = my_func(np.array(X_train))
But I get the following error:
AttributeError: 'DataFrame' object has no attribute 'convert_to_tensor'
I have no idea how to solve this. Any suggestion would be appreciated.
>Solution :
For some reason your X_train is composed of lists, it’s not an array.
You must make it a single array.
If all lists have the same size, you should be able to make something like
X_train = numpy.array([obj[0] for obj in X_train])
Make sure the shape is 2D with X_train.shape.
Another option, depending on how X_train is structured:
X_train = numpy.array([obj for obj in X_train])
The option that gives you a 2D shape and a X_train.dtype equals to some float is the right one.