2022-02-21, 13:28
(This post was last modified: 2022-02-21, 13:33 by ThunderStorm.)
Made some progress in building a program with python to estimate the polarities of detected strikes.
This is the very first version and it's quite rudimentary. Basically, for each strike, I get the signals of the detectors listed in the strike data, and try to make a match.
However, only about 20% of the strikes can be matched with signal data (from data.lightningmaps.org). I read here that it's a problem with the timestamps within the strikes or signals. Has anyone a working solution on how to time-match the two datasets?
Anyway, for the strikes that can be match, I filter only the signals with an E-field channel, then using a peak detection algorithm, I check for the sign of the first peak (from what I read in literature and from what was posted in this topic, so if the sign is -, the polarity is positive and vice-versa, as we are interested in the second peak sign, which is usually, the opposite of the first peak .. will work on this, to check for this duality between first and second peak, and to discard the analysis if it's too ambiguous).
Here I have a lot of work to do, to improve the current algorithm and to make the estimate more reliable (some conditionals, better peak detection functions, noise ignoring and so on).
Ok, then I create a list for each strike, with sign data from each signal involved, and find out the dominant sign and assign the polarity.
I am currently downloading signal data for a couple of days, to have a large dataset to work with and will post updates.
I also have the capability to archive some signal data (have about 4.5 TB of HDD space on my server). I have two servers, the first, a xeon physical server that I own (40 core, 80 threads, 128 GB RAM), which I use for WRF-ARW numerical weather prediction and I can also use it for my experiments with lightning data and a cloud webserver that can also be used for data storage.
This is the very first version and it's quite rudimentary. Basically, for each strike, I get the signals of the detectors listed in the strike data, and try to make a match.
However, only about 20% of the strikes can be matched with signal data (from data.lightningmaps.org). I read here that it's a problem with the timestamps within the strikes or signals. Has anyone a working solution on how to time-match the two datasets?
Anyway, for the strikes that can be match, I filter only the signals with an E-field channel, then using a peak detection algorithm, I check for the sign of the first peak (from what I read in literature and from what was posted in this topic, so if the sign is -, the polarity is positive and vice-versa, as we are interested in the second peak sign, which is usually, the opposite of the first peak .. will work on this, to check for this duality between first and second peak, and to discard the analysis if it's too ambiguous).
Here I have a lot of work to do, to improve the current algorithm and to make the estimate more reliable (some conditionals, better peak detection functions, noise ignoring and so on).
Ok, then I create a list for each strike, with sign data from each signal involved, and find out the dominant sign and assign the polarity.
I am currently downloading signal data for a couple of days, to have a large dataset to work with and will post updates.
I also have the capability to archive some signal data (have about 4.5 TB of HDD space on my server). I have two servers, the first, a xeon physical server that I own (40 core, 80 threads, 128 GB RAM), which I use for WRF-ARW numerical weather prediction and I can also use it for my experiments with lightning data and a cloud webserver that can also be used for data storage.
Stations: 2959