Results 1 to 2 of 2

Thread: [Suggest] coordinate data storage or 10 to the -13 idiocy

  1. #1

    Post [Suggest] coordinate data storage or 10 to the -13 idiocy

    The recorded Resolution for the GPS is frankly unnecessarily high. By that I'm referring to the precision with which the data is stored. Anything less than 10 to the -6 is complete overkill for the purposes of plotting averaged points or determining some triangulation method.

    I was talking to a friend who programs and he thinks you are using a multi-byte float to represent the decimal portion of the coordinate before it is written out. This results in output in both the GPX and KML files that is of a meaningless precision. It also increases file size by a portion as well. From one of my GPX files -- wpt lat="43.9814833333333" lon="-75.9441166666667

    Below describes why.

    The earth's circumference at the equator according to Wikipedia is 40,075.16 km. Since the earth is a flattened sphere points along the equator form the worst case scenario for distance between adjacent points.

    40,075.16 km / 360 degreees = 111.31988... km/degree

    Degrees are recorded separately so we can just drop them from the issue and just look at the divisions within a single degree. Under the old DMS nomenclature:

    111.319888 km / 3600 angular seconds = 0.0309221 km/second = 30.922191 meters/angular second(this is a distance, not a velocity)

    GPS units report a combined Degrees with decimal MinutesSeconds(MS) reading. This is what inSSIDer and other tracking programs use and log.
    For those of us in the US still using inches, it is 2.54 cm to the inch.
    Resolution		Position difference between adjacent points.
    x(or just degrees)	111.319888 km
    x.1			11.1319888 km
    x.01			1.11319888 km
    x.001			111.319888 m
    x.0001			11.1319888 m
    x.00001			1.11319888 m
    x.000001		11.1319888 cm ( about 4.25 inches)
    x.0000001		1.11319888 cm
    x.00000001		1.11319888 mm
    The point of all of this is at best WAAS data is accurate from 0.9 meters to 1.5 meters based on real world testing(see waas on wikipedia). GPS based Surveying equipment may be accurate down to 10-6 digits, but that gear costs about 8 grand a unit. Not something that a user of this application is likely to be using.

    Even if we lived on Jupiter, 10 to the -8 digits would define surface points separated by 1.247 cm.

    I would look at changing it myself if I could program at all. I'll be posting a few other suggestions each under its own subject.

  2. #2


    I do agree that the 13 digit granularity is overkill and the we could probably get by with using just 6 digits.
    Currently, we use 64-bit floating point numbers to hold the lat & lon.

    We will look into this. Thanks.

Tags for this Thread


Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts