Decrease data usage with Blue hardware
#25
(2016-01-20, 02:22)kevinmcc Wrote:
(2016-01-17, 10:21)Bart Wrote:
(2016-01-16, 09:16)kevinmcc Wrote: I was thinking QuickLZ, LZO, LZ4, LZ4-HC, or zlib for the the compression. My first concern would be how much data needs compressed and how much cpu power the Blitzortung receivers have. Would they be able to compress the data fast enough. The second concern is the the power needed to decompress at the server end.  I agree some data is not very compressible, some is very easily compressible. I am still waiting for System Blue to launch so I can participate. Having not seen the data myself, I can not judge. Compression may not even be a worth while option.

I see a significant issue with those choices. These compression algorithms only become efficient if you build up data in a buffer for a while. It'd be quite efficient if you would lets say build up data one hour at a time and then all send it in bulk, but that'd delay the detection significantly. While I presume the target is to minimize latency. You really want stream compression algorithms for this sort of job, and there are a few near-ideal cases but these are bad to use in practice for data you *must have* due to their total lack for redundancy. If you're willing to sacrifice a few detections you could probably get away with it though.

You are assuming that you need a large collection of data to build a good dictionary for compression. If you use a predefined dictionary you do not need to time to build a dictionary as is done with typical compression methods. I am going to assume the Blitzertung data is quite predictable, in a predefined format, and good dictionary could be devised ahead of time. With a dictionary predefined the data being sent can be compressed and sent much quicker while still be very efficient in both time and size.
I'd say that heavily depends on the station environment, in a perfect vacuum far away from any other sources electromagnetic sources this would hold true; the real world is very different though. And yes, noise is a quantity you can statistically model, but the exact parameters of said model are quite location dependant. So unless you manage to throw out most of the noise this generally wouldn't work very well. Additionally your main overhead would remain: the packet headers for networking. That's why stream compression has a chance of working though, but it must be applied correctly (increase signal delay and buffer data).
Reply


Messages In This Thread
RE: Decrease data usage with Blue hardware - by Bart - 2016-02-04, 20:55

Forum Jump:


Users browsing this thread: 1 Guest(s)