pyPhotometry data¶
pyPhotometry can save data either as binary files with a .ppd file extension or as comma seperated value files with a .csv file extension. File names are determined by the subject ID, date and time the recording started, and file type, e.g. m1-2018-08-30-103945.ppd
Saving the data as a binary .ppd file generates a single file per recording which contains both the acquisition settings and the data.
Saving the data as a .csv file generates two files per recording; a .csv file containing the data and a .json file containing the acquisition settings.
The binary data files are more compact than the .csv files, a 1 hour recording at 130Hz sampling rate yields a .ppd file of ~1.8MB and a .csv file of ~8MB.
Importing data¶
If you are using Python for analysis you can import .ppd files using the import_ppd function in the data_import module:
from data_import import import_ppd
data = import_ppd('path\\to\\data_file.ppd', low_pass=20, high_pass=0.001)
The import_ppd function returns a dictionary with the following entries:
'subject_ID' # Subject ID
'date_time' # Recording start date and time (ISO 8601 format string)
'mode' # Acquisition mode
'sampling_rate' # Sampling rate (Hz)
'LED_current' # Current for LEDs 1 and 2 (mA)
'version' # Version number of pyPhotometry
'analog_1' # Raw analog signal 1 (volts)
'analog_2' # Raw analog signal 2 (volts)
'analog_3' # Raw analog signal 3 (if present) (volts)
'analog_1_filt' # Filtered analog signal 1 (volts)
'analog_2_filt' # Filtered analog signal 2 (volts)
'digital_1' # Digital signal 1
'digital_2' # Digital signal 2
'pulse_inds_1' # Locations of rising edges on digital input 1 (samples).
'pulse_inds_2' # Locations of rising edges on digital input 2 (samples).
'pulse_times_1' # Times of rising edges on digital input 1 (ms).
'pulse_times_2' # Times of rising edges on digital input 2 (ms).
'time' # Time of each sample relative to start of recording (ms)
The high_pass and low_pass arguments provided to the import_data function determine the frequency in Hz of highpass and lowpass filtering applied to the filtered analog signals. To disable highpass or lowpass filtering set the respective argument to None. The filtering applies a 2nd order Butterworth filter in the forward and reverse directions to give a 4th order zero phase filter.
If you are using Matlab for analysis you can import data with the function import_ppd.m in the tools folder:
data = import_ppd('path\\to\\data_file.ppd')
The Matlab import function returns a struct with the same fields as the dictionary returned by the Python import function, but without the filtered versions of the analog signals or digital input pulse times.
Data preprocessing¶
Photometry data typically needs preprocessing to remove noise, and correct for photobleaching and movement artifacts. Some photometry data preprocessing methods are shown in this notebook. A Python function implementing the preprocessing method shown in the notebook is provided in the data import module. To use it do:
from data_import import import_ppd, preprocess_data
data = import_ppd('path\\to\\data_file.ppd')
processed_signal = preprocess_data(data_dict=data,
signal="analog_1",
control="analog_2",
low_pass=10,
normalisation="dF/F",
plot=True)
For more information see the function docstring.
Binary data format¶
The binary .ppd files generated by pyPhotometry have the following structure:
| Start byte | End byte | Content | Format |
|---|---|---|---|
| 1 | 2 | header size | 2 byte little endian integer |
| 3 | 2+header size | header | JSON object encoded as UTF-8 string |
| 3+header size | file end | data | see below |
The first two bytes of the file indicate the size of the header. The header is a UTF-8 encoded string which represents a JSON object with the following entries:
'subject_ID' # Subject ID
'date_time' # Recording start date and time (ISO 8601 format string)
'mode' # Acquisition mode
'sampling_rate' # Sampling rate (Hz).
'version' # Version number of pyPhotometry
'volts_per_division' # Volts per division of the analog signals.
'LED_current' # Current for LEDs 1 and 2 (mA)
The remainder of the data file contains the analog and digital signals. Each two bytes chunk of data encodes a 16 bit little endian unsigned integer. The most significant 15 bits of each integer encode one sample of an analog signal and the least significant bit encodes one sample of a digital signal. Analog channel 1 and digital channel 1 are paired together, and analog channel 2 and digital channel 2 are paired together. Channels 1 and 2 alternate in successive 2 byte chunks. To convert the analog signals to volts, multiply them by the 'volts_per_division' value from the header information.
In Python, the steps to convert the data bytes into signals are:
# Convert the data bytes into an array of 16 bit unsigned integers.
data = numpy.frombuffer(data_bytes, dtype=numpy.dtype('<u2'))
# Analog signals are most significant 15 bits of each integer,
# extract them by bit shifting 1 to the right.
analog = data >> 1
# Digital signals are least significant bit of each integer,
# extract them by bitwise AND with integer 1.
digital = data & 1
# Channels 1 and 2 are alternating samples:
analog_1 = analog[0::2]
analog_2 = analog[1::2]
digital_1 = digital[0::2]
digital_2 = digital[1::2]
# Convert the analog signals into volts.
analog_1 = analog_1 * volts_per_division[0]
analog_2 = analog_2 * volts_per_division[1]
Comma seperated value data format¶
The .csv files generated by pyPhotometry are UTF-8 encoded text files with 4 entries per line, seperated by commas. Each line contains one sample each from the two analog inputs and two digital inputs, in the order:
Analog_1, Analog_2, Digital_1, Digital_2
Each analog sample is an integer between 0 and 32768 and each digital sample an integer 0 or 1. The first line of the file contains the column names seperated by commas, such that the start of a file might read:
Analog1, Analog2, Digital1, Digital2
25443,13364,0,0
25435,13563,0,1
25442,13759,1,0
The .json file containing the acquisition settings is a UTF-8 encoded text file which represents a JSON object containing the same information as the binary data files header. The .json files are human readable if opened in a text editor. The analog signal values in the .csv file can be converted into volts using the volts_per_division information in the .json file.
Synchronisation¶
To synchronise pyPhotometry data with behavioural data we typically send sync puses from the behavioural hardware to a pyPhotometry digital input. For more information see the synchronisation page of the pyControl docs.
An example analysis showing how to synchronise pyControl behavioural data with neural activity recorded using pyPhotometry is provided in this data synchronisation jupyter notebook.