CSV logger plugin¶

CSV¶

A Comma Separated Value (CSV) file is a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas. A CSV file typically stores tabular data (numbers and text) in plain text, in which case each line will have the same number of fields.

This plugin connects to the uBrigde server, subscribes for all the available sensor's data, and record the incoming data into CSV files.

Tip: Increasing the reporting period

In order to keep the number CSV records into manageable sizes, the plugin configuration offers an option to limit the number of maximum datapoints per file (when this limit is hit, a new file is created).

Nevertheless, if the intention is to log data for very long periods of time, while keeping the data into a single file, an alternative is to increase the "reporting period" of the uThing device before connecting it to the uBridge application. Please check the configuration for each specific device (remember to save the configuration so it will be persistent by issuing the {"saveConfig": true} command).

CSV Files location¶

The logs path is set in the configuration file, see below. By default, the files are stored in /tmp/uThingData/.

File name¶

The file name is composed by the sensor type (i.e. uThingVOC), plus the last 4 characters of its serial number, plus the date and time of the creation of the file (in standard ISO8601 UTC time).

For example: uThingVOC_F8B6_2021-05-05_10-41-55Z.csv

Data format¶

As is often used in CSV files, the first line (record) indicates the fields (columns) meaning and order. The plugin detects the incoming data format and generate the header accordingly. The timestamp is always added at the end (see the configuration file for the format).

Example:

IAQ,eqBreathVOC,eqCO2,gasResistance,humidity,iaqAccuracy,pressure,temperature,timestamp
38.0,0.57,532.72,407383,27.0,3,991.88,29.66,2021-05-05T10:41:55Z
38.4,0.57,533.83,407383,26.99,3,991.84,29.67,2021-05-05T10:41:58Z

Timestamp format¶

The plugin provides 3 alternatives for the format (date-time in UTC or local-time, or Unix Epoch time in milliseconds). See the configuration file for more information.

Source code and Installation¶

The source code and installation instructions can be found here.

Configuration file¶

The configuration file is installed here: /etc/ubridge/csv-plugin-config.json

Here is the default configuration:

{
    "ubridge": /* socket configuration for the uBridge side: */
    {
        "configSockUrl" : "ipc:///tmp/ubridgeReqResp",/* The sockets for the Request Response Server and data Streamer*/
        "streamSockUrl" : "ipc:///tmp/ubridgeStream"/* If the client app runs in the same machine, IPC is 
                        more efficient. If a different host is used (as with a client running in a Docker instance)
                        use TCP host:port ("tcp://localhost:8001")*/
    },
    "csv":  /* configuration for the InfluxDB side: */
    {
        "logPath" : "/tmp/uThingData",/* The base path for storing the CSV files, change to "/home/$USER/$(FINAL_PATH)*/
        "maxDatapointsPerFile" : 100000,/* Limit the logged data to a manageable size
                                    when this limit is reached, a new file is created */
        "useGmtTime" : true,/* if false will use localtime*/
        "outTimestampInMs" : false/* default timestapmp format is unix epoch in milliseconds, 
                            if false use date-time (UTC or local depending on "useGmtTime" */
    }
}

"configSockUrl and streamSockUrl": These have to be the same as in the uBridge configuration.
"maxDatapointsPerFile": Limit the logged data to a manageable size. When this limit is reached, a new file is created.
"useGmtTime": When "true", uses GMT time (UTC). If "false" is used, the plugin will use the local time.
"outTimestampInMs": When "true", the timestamp is logged into Unix Epoch Time in milliseconds (number of milliseconds elapsed since 1 January 1970). If "false" is used, the plugin will use date-time format (with UTC or local depending on the "useGmtTime" configuration).

NOTE: Remember to restart the service after changing the configuration file (sudo systemctl restart ubridge-csv-plugin.service).

Importing in Excel / Google Sheets¶

If everything went well (ubridge and CSV plugin services running), when a "uThing" device is plugged into the USB port, the data should start flowing and CSV files will be created on the configured location.

Tip: Downloading files from a different machine

A quick way of downloading CSV files into a different host is to run python3 -m http.server from the location of the files (i.e. cd /tmp/uThingData/ && python3 -m http.server). This will open a webserver exposing the files on that directory. The port is 8000 by default.

Then, from a browser point to that server (i.e. http://raspberypi.local:8000), and a list of the available files will be available for downloading:

Then, importing the CSV is straightforward, with most Sheets applications supporting CSV import out-of-the-box. The capture below shows an example with Google Sheets:

Depending on the application used for import, you might want to set the "outTimestampInMs" configuration to true, since it might simplify the analysis without needing to parse the timestamp.

Tip: converting unix timestamp to date-time

If you generated the file with "outTimestampInMs : true", this handy formula for Excel converts the timestamp into date-time format: "= A1 * 0.001/86400 + date(1970,1,1)"

Monitoring¶

Check running status of the service¶

 systemctl status ubridge-csv-plugin.service

Log file¶

The log files for the uBridge and plugins are located in the /tmp directory:

tail /tmp/ubridge-csv.log