Forums

Massive problems reading pickled files saved under python 3.6 env. This is how we saved the files in python 3.6.

with open('sonde_adam_final.pkl', 'wb') as f:
    pickle.dump(sonde_data, f,protocol=2)

1st we tried with default protocol which defaults to version 3, then we tried protocol=2 - still no good.

This is how we try to open in python 3.6 env

sonde_data = pickle.load(open('sonde_adam_final.pkl', 'rb'), fix_imports=True, encoding='bytes')
Error: ModuleNotFoundError: No module named 'pandas.core.indexes'

sonde_data = pickle.loads(open('sonde_adam_final.pkl', 'rb'), fix_imports=True, encoding='bytes')
Error: TypeError: a bytes-like object is required, not '_io.BufferedReader'

sonde_data = pd.read_pickle(open('sonde_adam_final.pkl', 'rb''))
Error: TypeError: expected str, bytes or os.PathLike object, not _io.BufferedReader

sonde_data = pd.read_pickle('/home/vinorda/storm_predict/data/sonde_adam_final.pkl')
Error: ModuleNotFoundError: No module named 'pandas.core.indexes'

with open('/home/vinorda/storm_predict/data/sonde_adam_final.pkl', 'rb') as savefile:
    sonde_data = pd.read_pickle(savefile)
Error: TypeError: expected str, bytes or os.PathLike object, not _io.BufferedReader

with open('/home/vinorda/storm_predict/data/sonde_adam_final.pkl', 'rb') as savefile:
    sonde_data = pickle.load(savefile)
Error: ModuleNotFoundError: No module named 'pandas.core.indexes'

appreciate any help...

[edited by admin: formatting]

Some thoughts:

1. how did you make the pickle file? on your local machine or on PythonAnywhere?
2. module not found errors just means that you have not imported that module yet in your PythonAnywhere code
3. the other errors sounds like coding errors (eg: probably need something like pd.read_pickle(savefile.read()))

Thanks Conrad, The pickled file is just a simple pandas dataframe, which I had initially saved in Python 3.6 environment using picklle.dump(), without specifying any protocol like this

with open(data_dir+'f160/sonde_adam_final.pkl', 'wb') as f: pickle.dump(sonde_data, f)

On my machine (fedora 27 with Python 3.6) I can open it like this no problems.

with open(data_dir+'f160/sonde_adam_final.pkl', 'rb') as savefile: sonde_data = pickle.load(savefile)

I have read other blog suggesting I should try with open(data_dir+'f160/sonde_adam_final.pkl', 'rb') as savefile: sonde_data = pd.read_pickle(savefile)

I have tried the variant of above as per your suggestion and using absolute path

with open('/home/vinorda/storm_predict/data/sonde_adam_final.pkl', 'rb') as savefile: sonde_data = pd.read_pickle(savefile.read())

but that throws new error ValueError: embedded null byte File "/home/vinorda/storm_predict/flask_app.py", line 162, in <module> sonde_data = pd.read_pickle(savefile.read()) File "/usr/local/lib/python3.6/dist-packages/pandas/io/pickle.py", line 68, in read_pickle return try_read(path, encoding='latin1') File "/usr/local/lib/python3.6/dist-packages/pandas/io/pickle.py", line 61, in try_read with open(path, 'rb') as fh:

Possible causes (not a lot to do with my code but the ENVironment!!!) - One of the AppConfig objects has null byte in its path attribute. -One of the LOCALE_PATHS has null byte. -One of the files is in the "wrong" encoding, so Django treats something has null byte (e. g. AppConfig.path). - One of the files or directories in project directory has null byte (\x00) in its name.

I'll try encoding to protocol 3 and same errors

with open('/home/vinorda/storm_predict/data/YBSU_aws.pkl', 'rb') as savefile: #protocol 3 sonde_data = pd.read_pickle(savefile.read()) ValueError: embedded null byte

with open('/home/vinorda/storm_predict/data/YBSU_aws.pkl', 'rb') as savefile: sonde_data = pd.read_pickle(savefile)) TypeError: expected str, bytes or os.PathLike object, not _io.BufferedReader

I can't really tell from all that what the issue might be. Perhaps you can use pickletools to work out what is going on.

hey vinorda, i think i have the solution for your code. here is my code for unpickling the pickle :

file_name= "ezra tweets.pkl"
with open(os.path.join(my_dir, file_name), 'wb') as f:
    pickle.dump(dfTweets, f,protocol=2)

it seems like the problem lies in how you access the file. In this code, the pickle file is saved to mysite directory where the .py code is

[edit by admin: formatting]