Decrease data usage with Blue hardware

[anakaine]

Hi, 

I've been following blitzortung as an observer for a little while, and have the ability to add a sensor in central QLD, Australia. This would help cover Papua New Guinea, Noumea, and north east Australia better, where no sensors exist. The question that I have, however, is can the data usage be decreased? Currently the network available uses a 3G backhaul with a 6gb monthly allowance and is shared by 15+ users. We will likely be going to a satellite connection, however this will also be limited - perhaps 30-40gb / month. I see the Red board can pump out 3gb+ a day during intense storms - and being proximal to the tropics during the wet season this site could expect to see some of the most intense electrical storms on the planet. 

Is there a method where data can be compressed, and usage lowered?

[Tobi]

Hi,

yes, the high amount of traffic is an issue that we want to solve, i.e. by transmitting only usable data/channels. There's currently work in progress. It's independent from the system (RED/BLUE) as it's only related to the data transmission. A new (beta) firmware will be out very soon which will address even more issues. Please be patient.

Best regards
Tobias

[kevinmcc]

Is the data uncompressed? Compression could be an option.

[Steph]

I'd estimate with lossless compression there couldn't be saved much more than 50% of the traffic, which probably wouldn't be enough for anakaine. However it's possible to lower the gain, detect less strikes and thus have less traffic.

But another option came to my mind; maybe it's possible in exceptional cases to only transmit the timestamp and some parameters instead of the whole wavelet.
But that's a thing Tobi or Egon have to confirm whether that's an option, since the waveform may be taken into account for calculation at some point.

[Tobi]

I'd estimate with lossless compression there couldn't be saved much more than 50% of the traffic, which probably wouldn't be enough for anakaine. However it's possible to lower the gain, detect less strikes and thus have less traffic.

[/quote]
It's even less, maybe 10 to 20%. It's binary data like exe or jpeg, which can not be compressed very much.

But another option came to my mind; maybe it's possible in exceptional cases to only transmit the timestamp and some parameters instead of the whole wavelet.
But that's a thing Tobi or Egon have to confirm whether that's an option, since the waveform may be taken into account for calculation at some point.

Yes, that's what I'm currently working on, but the timestamp itself is not enough for accurate location calculation.

[Steph]

It's even less, maybe 10 to 20%. It's binary data like exe or jpeg, which can not be compressed very much.

I agree, if you use simple "dictionary"-type of compression on the original signal.
But if you only store the deltas between each sample you can save some bits, since there is not much change between the samples.

For example: If i store the signal from my green station it's 256 Bytes, with default gzip it's 170 Bytes.
But if i do a delta between each sample I only need 2 or 3 bits per sample. So it's resulting in an uncompressed 64-96 Bytes.

For red stations, or blue with higher frequencies it's probably bigger, but i still think 50% is archievable, or even more if you find a way to use dynamic bits per sample.

[Bart]

To drop in my two cents: I have to agree with Steph, a delta encoder has a good chance of performing well if you actually send a waveform.

And actually several other stream compression methods could potentially perform quite well as well. Additionally the question is which sort of fidelity is necessary, if you know what you really need in the signal you could potentially build a short period of the signal out of a series of pre-determined orthogonal elements that are linearly combined in a way to match the signal with an acceptable error. Sending that combination could be a lot "cheaper" in terms of data than the current method. Question is of course how feasible it'd be to do that one on the fly with the existing hardware?

There are three issues with most methods I can think of though:
a) Server load during busy moments,
b) The moment you lose one packet you could probably throw away the rest of the data if you don't add redundancy.
c) Latency (if you hold on to your data longer you could gain more but you'll lose the quick response time).

[kevinmcc]

I was thinking QuickLZ, LZO, LZ4, LZ4-HC, or zlib for the the compression. My first concern would be how much data needs compressed and how much cpu power the Blitzortung receivers have. Would they be able to compress the data fast enough. The second concern is the the power needed to decompress at the server end. I agree some data is not very compressible, some is very easily compressible. I am still waiting for System Blue to launch so I can participate. Having not seen the data myself, I can not judge. Compression may not even be a worth while option.

[cutty]

The majority of the data transmitted is 'junk' .  High gains, many useless Skywaves and Noise signals, local disturbers (which I am plagued with). 
 The goal is to eliminate as much junk as possible.  Ideally, perhaps, the network would work with stations covering radius of <1000km radius, and not attempting to detect signals in other hemispheres.  This requires station density and optimizations.  Many stations send too much noise, including mine.  As more stations come on line, those such as I will be reducing our antennas and gain settings.  Some of those stations are quite clean, in a good environment, and do very well with longer range. Many do not.
There will be a push for smaller antennas and less 'distance' capability, quality of data, etc. Since that is NOT the way the system is envisioned.  These are not 'stand alone' systems, but must participate as a 'cell' in a network to be effective.
A local station should normally go interference, and quit transmitting with 'nearby' storms... the rest of the network picks up the data, for example.
The TOA / TOGA system considers the whole pulse train, the frequencies and 'respective energy'  contained in the impulse, and not just the discharge pulse timing at trigger, or triggering by 1st or 2nd skywave signal... therefore 'recreating' or 'interpolating'  those zero crossing iterations from a 'sampling' would likely result in 'distortion' of the quality control and stroke information. The system wants as much of the complete stroke, with real data, as clean as possible.

[weatherc]

First, i agree on trying to get down the feeded data-amount as much as possible. Not all have unlimited connections and not all, including mine, have high-speed connections either.
Could one idea be to have some kind of "low traffic" setting as option where it reduces the feeded data-amount to a minimum even if the hitrate decreases a bit?

As more stations come on line, those such as I will be reducing our antennas and gain settings.

That sounds good only on paper. Any such thing need to be forced from the server-end if want it really to applied, a "can you please lower your gain"-thing will never work. It can be well seen on the weathernetworks (like EWN i admin) that not all care about their stations.