New style branch

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

New style branch

Steve Ratcliffe
Hi

Every now and again, there is a bug report on a style rule that gives
an error when it shouldn't.  The most recent from Mike Baggaley,
included a case where the rule was accepted but the result was
incorrect.

I wrote a little program to generate random rules and test them using
the style tester.

This showed that there were many rules that should be accepted that
are not and worse there are cases where the wrong result is given.

Worst of all this simple rule:

    a>=1 & (a=1 | b~2) [0x2]

(and many others) crash mkgmap with a StackOverflowError.  I'm
surprised that no-one has found that.  Perhaps they have and it just
hasn't been reported.

An example of a rule that is accepted but doesn't work is:

    (a!=1 | b~2) & a<2 & a>=2 [0x2]

This matches (with trunk) with a way with tags a=2 and b=2
but it should not.

So not very complex rules fail.  There are also vast number of rules
that involve using not (!) such as !(highway=service & ...) that are
not accepted or do not work.  Probably not many styles have many such
rules though.

So I re-wrote and simplified the code and the result is
on the expr branch.

There is also a new feature that was needed to help make it
work.  There is now a not regex match operator !~.  It is
probably not all that useful but it is now availabl.

    name !~ 'Main.*'

is exactly the same as

    !(name ~ 'Main.*')


The latest branch build can be downloaded from:

    http://www.mkgmap.org.uk/download/mkgmap-expr-latest.zip

It is important to test it, as although I believe that the branch is
now correct, some styles may have been relying on previous bugs.

..Steve
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: New style branch

Felix Hartmann-2
I ran into this bug many years ago - but back then I either assumed or got told that the first rule must not be != but has to be plain =.
However I think it simply did not work instead of throwing a stack overflow.

On 7 August 2017 at 22:43, Steve Ratcliffe <[hidden email]> wrote:
Hi

Every now and again, there is a bug report on a style rule that gives
an error when it shouldn't.  The most recent from Mike Baggaley,
included a case where the rule was accepted but the result was
incorrect.

I wrote a little program to generate random rules and test them using
the style tester.

This showed that there were many rules that should be accepted that
are not and worse there are cases where the wrong result is given.

Worst of all this simple rule:

   a>=1 & (a=1 | b~2) [0x2]

(and many others) crash mkgmap with a StackOverflowError.  I'm
surprised that no-one has found that.  Perhaps they have and it just
hasn't been reported.

An example of a rule that is accepted but doesn't work is:

   (a!=1 | b~2) & a<2 & a>=2 [0x2]

This matches (with trunk) with a way with tags a=2 and b=2
but it should not.

So not very complex rules fail.  There are also vast number of rules
that involve using not (!) such as !(highway=service & ...) that are
not accepted or do not work.  Probably not many styles have many such
rules though.

So I re-wrote and simplified the code and the result is
on the expr branch.

There is also a new feature that was needed to help make it
work.  There is now a not regex match operator !~.  It is
probably not all that useful but it is now availabl.

   name !~ 'Main.*'

is exactly the same as

   !(name ~ 'Main.*')


The latest branch build can be downloaded from:

   http://www.mkgmap.org.uk/download/mkgmap-expr-latest.zip

It is important to test it, as although I believe that the branch is
now correct, some styles may have been relying on previous bugs.

..Steve
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev



--
Felix Hartman - Openmtbmap.org & VeloMap.org
Schusterbergweg 32/8
6020 Innsbruck
Austria - Österreich

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: New style branch

Gerd Petermann
In reply to this post by Steve Ratcliffe
Hi Steve,

I've tried it with a slightly modified version of Minkos OFM lite style and got a NPE.
My command:
java -jar c:\temp\mkgmap-expr-r3990\mkgmap.jar  --style-file=c:\mystyles\ofm_lite --check-styles
Style see http://files.mkgmap.org.uk/download/355/ofm_lite.zip
I've also noted an experimental line in the default style lines file:
!!(aeroway=runway & highway!=*) & foo=bar [0x1]

Gerd
java.lang.NullPointerException
        at uk.me.parabola.mkgmap.osmstyle.eval.NotExistsOp.toString(NotExistsOp.java:39)
        at uk.me.parabola.mkgmap.osmstyle.eval.AbstractBinaryOp.toString(AbstractBinaryOp.java:54)
        at uk.me.parabola.mkgmap.osmstyle.eval.AbstractBinaryOp.toString(AbstractBinaryOp.java:65)
        at java.util.Formatter$FormatSpecifier.printString(Unknown Source)
        at java.util.Formatter$FormatSpecifier.print(Unknown Source)
        at java.util.Formatter.format(Unknown Source)
        at java.util.Formatter.format(Unknown Source)
        at java.lang.String.format(Unknown Source)
        at uk.me.parabola.mkgmap.osmstyle.ExpressionArranger.fmtExpr(ExpressionArranger.java:560)
        at uk.me.parabola.mkgmap.osmstyle.ExpressionArranger.arrange(ExpressionArranger.java:73)
        at uk.me.parabola.mkgmap.osmstyle.RuleFileReader.loadFile(RuleFileReader.java:125)
        at uk.me.parabola.mkgmap.osmstyle.RuleFileReader.load(RuleFileReader.java:95)
        at uk.me.parabola.mkgmap.osmstyle.StyleImpl.readRules(StyleImpl.java:301)
        at uk.me.parabola.mkgmap.osmstyle.StyleImpl.<init>(StyleImpl.java:155)
        at uk.me.parabola.mkgmap.osmstyle.StyleImpl.readStyle(StyleImpl.java:545)
        at uk.me.parabola.mkgmap.reader.osm.OsmMapDataSource.createConverter(OsmMapDataSource.java:333)
        at uk.me.parabola.mkgmap.reader.osm.OsmMapDataSource.setupHandler(OsmMapDataSource.java:226)
        at uk.me.parabola.mkgmap.reader.osm.OsmMapDataSource.parse(OsmMapDataSource.java:168)
        at uk.me.parabola.mkgmap.reader.osm.OsmMapDataSource.load(OsmMapDataSource.java:150)
        at uk.me.parabola.mkgmap.main.MapMaker.loadFromFile(MapMaker.java:154)
        at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:52)
        at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:263)
        at uk.me.parabola.mkgmap.main.Main$1.call(Main.java:259)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von Steve Ratcliffe <[hidden email]>
Gesendet: Montag, 7. August 2017 22:43:55
An: mkgmap development
Betreff: [mkgmap-dev] New style branch

Hi

Every now and again, there is a bug report on a style rule that gives
an error when it shouldn't.  The most recent from Mike Baggaley,
included a case where the rule was accepted but the result was
incorrect.

I wrote a little program to generate random rules and test them using
the style tester.

This showed that there were many rules that should be accepted that
are not and worse there are cases where the wrong result is given.

Worst of all this simple rule:

    a>=1 & (a=1 | b~2) [0x2]

(and many others) crash mkgmap with a StackOverflowError.  I'm
surprised that no-one has found that.  Perhaps they have and it just
hasn't been reported.

An example of a rule that is accepted but doesn't work is:

    (a!=1 | b~2) & a<2 & a>=2 [0x2]

This matches (with trunk) with a way with tags a=2 and b=2
but it should not.

So not very complex rules fail.  There are also vast number of rules
that involve using not (!) such as !(highway=service & ...) that are
not accepted or do not work.  Probably not many styles have many such
rules though.

So I re-wrote and simplified the code and the result is
on the expr branch.

There is also a new feature that was needed to help make it
work.  There is now a not regex match operator !~.  It is
probably not all that useful but it is now availabl.

    name !~ 'Main.*'

is exactly the same as

    !(name ~ 'Main.*')


The latest branch build can be downloaded from:

    http://www.mkgmap.org.uk/download/mkgmap-expr-latest.zip

It is important to test it, as although I believe that the branch is
now correct, some styles may have been relying on previous bugs.

..Steve
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: New style branch

Steve Ratcliffe

Hi Gerd,

> I've tried it with a slightly modified version of Minkos OFM lite style and got a NPE.

Thanks for testing.  My random rules were not including any exists
nodes so didn't catch that one :-(

I've removed the experimental line from the lines file.

Cheers
Steve

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: New style branch

Gerd Petermann
Hi Steve,

one more small problem:
For the attached style the check-styles option says:
e:\ld>java -jar c:\temp\mkgmap-expr-r3991\mkgmap.jar --style-file=c:\mystyles\mini --check-styles
Time started: Fri Aug 11 08:09:08 CEST 2017
Found one style in c:\mystyles\mini
Error in style: Error: (polygons:9): Invalid rule expression: $building!=no & $building!=public & $building!=school & $building!=church & $building!=cathredral & $building!=chapel & $building!=proposed & $mkgmap:residential!=*
...

Would it be possible to report the 1st line of the rule (polygons:1) instead of the empty line after the wrong rule?
In this case the author first thought that this is related to an eol/ eof problem because the rule also appeared at the end of the file.

ciao,
Gerd
________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von Steve Ratcliffe <[hidden email]>
Gesendet: Mittwoch, 9. August 2017 12:58:43
An: [hidden email]
Betreff: Re: [mkgmap-dev] New style branch

Hi Gerd,

> I've tried it with a slightly modified version of Minkos OFM lite style and got a NPE.

Thanks for testing.  My random rules were not including any exists
nodes so didn't catch that one :-(

I've removed the experimental line from the lines file.

Cheers
Steve

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev