coshp utility to support "cloud optimized shape file"

[pramsey]

To enable fast bounding box access of shapes over HTTP-offset reads, we need to sort the contents of shape files so that they appear in the same order in the shp file as they do in the qix spatial index. (Also the contents of the dbf need similar sorting.

In the spirit of sortshp the coshp utility takes in an unsorted shape file, and writes out a sorted shape file (shp, shx, dbf, qix).

> coshp                                   
Syntax: coshp [infile] [outfile]

> coshp va_2013_carto va_2013_carto_sorted
Wrote 9349 spatially sorted shapes into 'shapefile va_2013_carto_sorted'

[jmckenna]

Thanks @pramsey . Is it possible for you to also include an msautotest/test as part of this PR for this utility? (see https://mapserver.org/development/tests/autotest.html )

[pramsey]

This doesn't seem to be standard practice for the utility programs? I cannot find any tests running for shptree or sortshp.

[jmckenna]

I'm not sure about 'standard practice', but I think for a new utility as part of the MapServer 8.0 release the new utility should be tested as part of msautotest.

Hope that answers your question.

[rouault]

I cannot find any tests running for shptree or sortshp.

true, we are lacking on that side.
Potential way forward:

• add a msautotest/utils directory with a 0-byte __init__.py file
• referencing that new directory in testpaths of msautotest/pytest.ini
• add a msautotest/utils/test_coshp.py file importing pytest, having one or several test functions (prefixed by "test_") and using os.system() to run coshp and doing some after checks
• add a utils-testcase target in top Makefile that just cd msautotest/utils and run pytest in it

[sdlime]

I cannot find any tests running for shptree or sortshp.

true, we are lacking on that side. Potential way forward:

• add a msautotest/utils directory with a 0-byte __init__.py file
• referencing that new directory in testpaths of msautotest/pytest.ini
• add a msautotest/utils/test_coshp.py file importing pytest, having one or several test functions (prefixed by "test_") and using os.system() to run coshp and doing some after checks
• add a utils-testcase target in top Makefile that just cd msautotest/utils and run pytest in it

Seems like reasonable tests would be:

For sortshp and coshp:

1. compare a source and output shapefile properties e.g. feature count, extent.
2. compare maps rendered from source and output shapefiles.

For shptree:

1. compare maps rendered from a source shapefile with and without an index.

I'd can take a shot at @rouault suggestion for sortshp I guess.

[jmckenna]

Should I tackle the msautotests for this? (beta release will be this coming Friday) Thanks!

[sdlime]

IMHO, I don't think the tests should hold up the beta. I'd recommend a merge and then create a separate ticket related to the tests... --Steve

[jmckenna]

thanks @pramsey ! merging now...