JSON files with the pandas module

JSON strings or files can be read with the pandas.read_json() function, which returns a DataFrame or series object. For example, the following code reads the zips.json file:

df = pd.read_json(os.path.join(data_folder,data_file), lines=True)
print(df)

We set lines=True because each line contains a separate object in JSON format. Without this argument being set to True, pandas will raise ValueError. The DataFrame is printed as follows:

         _id             city                               loc    pop state
0       1001           AGAWAM           [-72.622739, 42.070206]  15338    MA
1       1002          CUSHMAN            [-72.51565, 42.377017]  36963    MA
...      ...              ...                               ...    ...   ...
29351  99929         WRANGELL          [-132.352918, 56.433524]   2573    AK
29352  99950        KETCHIKAN           [-133.18479, 55.942471]    422    AK

[29353 rows x 5 columns]

To save the pandas DataFrame or series object to a JSON file or string, use the Dataframe.to_json() function.

While CSV and JSON remain the most popular data formats for IoT data, due to its large size, it is often necessary to distribute data. There are two popular distributed mechanisms for data storage and access: HDF5 and HDFS. Let's first learn about the HDF5 format.