Building map with Hebrew characters

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Building map with Hebrew characters

Carlos Dávila-2
I'm trying to build a map of Israel with Hebrew characters, but using
--code-page=1255 a get the following assertion:

at uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:117)
         at
uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt4Chars(SRTFile.java:99)
         at uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:63)
         at
uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117)
         at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605)
         at
uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:128)
         at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
         at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

With --unicode I also get an assertion:
Exception in thread "main" java.lang.AssertionError: -125
         at
uk.me.parabola.imgfmt.app.BufferedImgFileWriter.put1u(BufferedImgFileWriter.java:154)
         at
uk.me.parabola.imgfmt.app.SectionWriter.put1u(SectionWriter.java:78)
         at
uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:113)
         at
uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt8(SRTFile.java:166)
         at uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:68)
         at
uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117)
         at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605)
         at
uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:128)
         at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
         at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

Probably I'm missing something obvious, but I always used
--code-page=1251 and have no idea what it may be

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Building map with Hebrew characters

nwillink
Have done some checking using older versions

It works on 4143 but not on 4179

I'm afraid I don't have anything in between

r

Nick

On 05/04/2019 21:25, Carlos Dávila wrote:

> I'm trying to build a map of Israel with Hebrew characters, but using
> --code-page=1255 a get the following assertion:
>
> at uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:117)
>         at
> uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt4Chars(SRTFile.java:99)
>         at uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:63)
>         at
> uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117)
>         at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605)
>         at
> uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:128)
>         at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
>         at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)
>
> With --unicode I also get an assertion:
> Exception in thread "main" java.lang.AssertionError: -125
>         at
> uk.me.parabola.imgfmt.app.BufferedImgFileWriter.put1u(BufferedImgFileWriter.java:154)
>         at
> uk.me.parabola.imgfmt.app.SectionWriter.put1u(SectionWriter.java:78)
>         at
> uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:113)
>         at
> uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt8(SRTFile.java:166)
>         at uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:68)
>         at
> uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117)
>         at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605)
>         at
> uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:128)
>         at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
>         at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)
>
> Probably I'm missing something obvious, but I always used
> --code-page=1251 and have no idea what it may be
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Building map with Hebrew characters

Gerd Petermann
Hi,
maybe a regression from r4167?
http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4167

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
Gesendet: Samstag, 6. April 2019 10:01
An: [hidden email]
Betreff: Re: [mkgmap-dev] Building map with Hebrew characters

Have done some checking using older versions

It works on 4143 but not on 4179

I'm afraid I don't have anything in between

r

Nick

On 05/04/2019 21:25, Carlos Dávila wrote:

> I'm trying to build a map of Israel with Hebrew characters, but using
> --code-page=1255 a get the following assertion:
>
> at uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:117)
>         at
> uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt4Chars(SRTFile.java:99)
>         at uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:63)
>         at
> uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117)
>         at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605)
>         at
> uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:128)
>         at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
>         at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)
>
> With --unicode I also get an assertion:
> Exception in thread "main" java.lang.AssertionError: -125
>         at
> uk.me.parabola.imgfmt.app.BufferedImgFileWriter.put1u(BufferedImgFileWriter.java:154)
>         at
> uk.me.parabola.imgfmt.app.SectionWriter.put1u(SectionWriter.java:78)
>         at
> uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:113)
>         at
> uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt8(SRTFile.java:166)
>         at uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:68)
>         at
> uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117)
>         at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605)
>         at
> uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java:128)
>         at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
>         at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)
>
> Probably I'm missing something obvious, but I always used
> --code-page=1251 and have no idea what it may be
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Building map with Hebrew characters

Ticker Berkin
I'm away from my development computer this weekend. I'll investigate on
Monday

Ticker

On Sat, 2019-04-06 at 08:16 +0000, Gerd Petermann wrote:

> Hi,
> maybe a regression from r4167?
> http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4167
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag
> von osm@pinns <[hidden email]>
> Gesendet: Samstag, 6. April 2019 10:01
> An: [hidden email]
> Betreff: Re: [mkgmap-dev] Building map with Hebrew characters
>
> Have done some checking using older versions
>
> It works on 4143 but not on 4179
>
> I'm afraid I don't have anything in between
>
> r
>
> Nick
>
> On 05/04/2019 21:25, Carlos Dávila wrote:
> > I'm trying to build a map of Israel with Hebrew characters, but
> > using
> > --code-page=1255 a get the following assertion:
> >
> > at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:117
> > )
> >         at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt4Chars(SRTFile.java:9
> > 9)
> >         at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:63)
> >         at
> > uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117
> > )
> >         at
> > uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605)
> >         at
> > uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.
> > java:128)
> >         at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
> >         at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)
> >
> > With --unicode I also get an assertion:
> > Exception in thread "main" java.lang.AssertionError: -125
> >         at
> > uk.me.parabola.imgfmt.app.BufferedImgFileWriter.put1u(BufferedImgFi
> > leWriter.java:154)
> >         at
> > uk.me.parabola.imgfmt.app.SectionWriter.put1u(SectionWriter.java:78
> > )
> >         at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:113
> > )
> >         at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt8(SRTFile.java:166)
> >         at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:68)
> >         at
> > uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117
> > )
> >         at
> > uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605)
> >         at
> > uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.
> > java:128)
> >         at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
> >         at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)
> >
> > Probably I'm missing something obvious, but I always used
> > --code-page=1251 and have no idea what it may be
> >
> > _______________________________________________
> > mkgmap-dev mailing list
> > [hidden email]
> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Building map with Hebrew characters

Ticker Berkin
In reply to this post by Gerd Petermann
Hi Gerd and Nick

@nick, when you say it works on 4143 but not on 4179, did you test code
-page=1255 on 4143, or just unicode?

It looks like there are 2 problems.

The --unicode assertion error is a 'byte' to 'int' sign-extension
problem, introduced in r4167 or r4168 and I've attached a patch that
fixes this and also removes some other confusing/obscuring type
representation changes in this area of the code.

The patch also has 4 commented out (because not appropriate for -
-unicode) asserts that seem to point to the Hebrew/1255 problem being
in imgfmt/app/srt/Sort.java:finish():

$ java -ea -jar /mkgmap/mkgmap.jar --code-page=1255 --gmapsupp test
-map:all-elements

Time started: Mon Apr 08 09:30:06 BST 2019
Exception in thread "main" java.lang.AssertionError: 16
        at
uk.me.parabola.imgfmt.app.srt.Sort$Page.setSecondary(Sort.java:642)
        at uk.me.parabola.imgfmt.app.srt.Sort.finish(Sort.java:161)
        at
uk.me.parabola.mkgmap.srt.SrtTextReader.read(SrtTextReader.java:182)
        at
uk.me.parabola.mkgmap.srt.SrtTextReader.<init>(SrtTextReader.java:117)
        at
uk.me.parabola.mkgmap.srt.SrtTextReader.<init>(SrtTextReader.java:106)
        at
uk.me.parabola.mkgmap.srt.SrtTextReader.sortForCodepage(SrtTextReader.j
ava:141)
        at uk.me.parabola.mkgmap.main.Main.getSort(Main.java:764)
        at
uk.me.parabola.mkgmap.main.Main.processFilename(Main.java:283)
        at
uk.me.parabola.mkgmap.CommandArgsReader$Filename.processArg(CommandArgs
Reader.java:272)
        at
uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java
:125)
        at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
        at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

this is corresponds to:
                                        } else {

                                                p.setSecondary(i,
p.getSecondary(i) + maxSecondary);
                                     
   }

where this increase in the secondary value is governed by some bits in the pages.flags and the flags testing was changed in r3968, along with the introduction of the original assert that was failing (SRTFile.java:117) and quite a few other changes in srt.

@gerd, I'm afraid I don't understand the srt logic so can you have a
look at it

Regards
Ticker

On Sat, 2019-04-06 at 08:16 +0000, Gerd Petermann wrote:

> Hi,
> maybe a regression from r4167?
> http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4167
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag
> von osm@pinns <[hidden email]>
> Gesendet: Samstag, 6. April 2019 10:01
> An: [hidden email]
> Betreff: Re: [mkgmap-dev] Building map with Hebrew characters
>
> Have done some checking using older versions
>
> It works on 4143 but not on 4179
>
> I'm afraid I don't have anything in between
>
> r
>
> Nick
>
> On 05/04/2019 21:25, Carlos Dávila wrote:
> > I'm trying to build a map of Israel with Hebrew characters, but
> > using
> > --code-page=1255 a get the following assertion:
> >
> > at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:117
> > )
> >         at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt4Chars(SRTFile.java:9
> > 9)
> >         at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:63)
> >         at
> > uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117
> > )
> >         at
> > uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605)
> >         at
> > uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.
> > java:128)
> >         at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
> >         at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)
> >
> > With --unicode I also get an assertion:
> > Exception in thread "main" java.lang.AssertionError: -125
> >         at
> > uk.me.parabola.imgfmt.app.BufferedImgFileWriter.put1u(BufferedImgFi
> > leWriter.java:154)
> >         at
> > uk.me.parabola.imgfmt.app.SectionWriter.put1u(SectionWriter.java:78
> > )
> >         at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:113
> > )
> >         at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt8(SRTFile.java:166)
> >         at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:68)
> >         at
> > uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117
> > )
> >         at
> > uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605)
> >         at
> > uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.
> > java:128)
> >         at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
> >         at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)
> >
> > Probably I'm missing something obvious, but I always used
> > --code-page=1251 and have no idea what it may be
> >
> > _______________________________________________
> > mkgmap-dev mailing list
> > [hidden email]
> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

charSet.patch (9K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Building map with Hebrew characters

Gerd Petermann
Hi Ticker,

sorry, can't help with this part of the source.

@Steve: Please help

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von Ticker Berkin <[hidden email]>
Gesendet: Montag, 8. April 2019 12:38
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] Building map with Hebrew characters

Hi Gerd and Nick

@nick, when you say it works on 4143 but not on 4179, did you test code
-page=1255 on 4143, or just unicode?

It looks like there are 2 problems.

The --unicode assertion error is a 'byte' to 'int' sign-extension
problem, introduced in r4167 or r4168 and I've attached a patch that
fixes this and also removes some other confusing/obscuring type
representation changes in this area of the code.

The patch also has 4 commented out (because not appropriate for -
-unicode) asserts that seem to point to the Hebrew/1255 problem being
in imgfmt/app/srt/Sort.java:finish():

$ java -ea -jar /mkgmap/mkgmap.jar --code-page=1255 --gmapsupp test
-map:all-elements

Time started: Mon Apr 08 09:30:06 BST 2019
Exception in thread "main" java.lang.AssertionError: 16
        at
uk.me.parabola.imgfmt.app.srt.Sort$Page.setSecondary(Sort.java:642)
        at uk.me.parabola.imgfmt.app.srt.Sort.finish(Sort.java:161)
        at
uk.me.parabola.mkgmap.srt.SrtTextReader.read(SrtTextReader.java:182)
        at
uk.me.parabola.mkgmap.srt.SrtTextReader.<init>(SrtTextReader.java:117)
        at
uk.me.parabola.mkgmap.srt.SrtTextReader.<init>(SrtTextReader.java:106)
        at
uk.me.parabola.mkgmap.srt.SrtTextReader.sortForCodepage(SrtTextReader.j
ava:141)
        at uk.me.parabola.mkgmap.main.Main.getSort(Main.java:764)
        at
uk.me.parabola.mkgmap.main.Main.processFilename(Main.java:283)
        at
uk.me.parabola.mkgmap.CommandArgsReader$Filename.processArg(CommandArgs
Reader.java:272)
        at
uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java
:125)
        at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
        at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

this is corresponds to:
                                        } else {

                                                p.setSecondary(i,
p.getSecondary(i) + maxSecondary);

   }

where this increase in the secondary value is governed by some bits in the pages.flags and the flags testing was changed in r3968, along with the introduction of the original assert that was failing (SRTFile.java:117) and quite a few other changes in srt.

@gerd, I'm afraid I don't understand the srt logic so can you have a
look at it

Regards
Ticker

On Sat, 2019-04-06 at 08:16 +0000, Gerd Petermann wrote:

> Hi,
> maybe a regression from r4167?
> http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4167
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag
> von osm@pinns <[hidden email]>
> Gesendet: Samstag, 6. April 2019 10:01
> An: [hidden email]
> Betreff: Re: [mkgmap-dev] Building map with Hebrew characters
>
> Have done some checking using older versions
>
> It works on 4143 but not on 4179
>
> I'm afraid I don't have anything in between
>
> r
>
> Nick
>
> On 05/04/2019 21:25, Carlos Dávila wrote:
> > I'm trying to build a map of Israel with Hebrew characters, but
> > using
> > --code-page=1255 a get the following assertion:
> >
> > at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:117
> > )
> >         at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt4Chars(SRTFile.java:9
> > 9)
> >         at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:63)
> >         at
> > uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117
> > )
> >         at
> > uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605)
> >         at
> > uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.
> > java:128)
> >         at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
> >         at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)
> >
> > With --unicode I also get an assertion:
> > Exception in thread "main" java.lang.AssertionError: -125
> >         at
> > uk.me.parabola.imgfmt.app.BufferedImgFileWriter.put1u(BufferedImgFi
> > leWriter.java:154)
> >         at
> > uk.me.parabola.imgfmt.app.SectionWriter.put1u(SectionWriter.java:78
> > )
> >         at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:113
> > )
> >         at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt8(SRTFile.java:166)
> >         at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:68)
> >         at
> > uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117
> > )
> >         at
> > uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605)
> >         at
> > uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.
> > java:128)
> >         at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
> >         at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)
> >
> > Probably I'm missing something obvious, but I always used
> > --code-page=1251 and have no idea what it may be
> >
> > _______________________________________________
> > mkgmap-dev mailing list
> > [hidden email]
> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Building map with Hebrew characters

nwillink
In reply to this post by Ticker Berkin

Hi Ticker

I just used unicode

r

Nick

On 08/04/2019 11:38, Ticker Berkin wrote:
Hi Gerd and Nick

@nick, when you say it works on 4143 but not on 4179, did you test code
-page=1255 on 4143, or just unicode?

It looks like there are 2 problems.

The --unicode assertion error is a 'byte' to 'int' sign-extension
problem, introduced in r4167 or r4168 and I've attached a patch that
fixes this and also removes some other confusing/obscuring type
representation changes in this area of the code.

The patch also has 4 commented out (because not appropriate for -
-unicode) asserts that seem to point to the Hebrew/1255 problem being
in imgfmt/app/srt/Sort.java:finish():

$ java -ea -jar /mkgmap/mkgmap.jar --code-page=1255 --gmapsupp test
-map:all-elements

Time started: Mon Apr 08 09:30:06 BST 2019
Exception in thread "main" java.lang.AssertionError: 16
        at
uk.me.parabola.imgfmt.app.srt.Sort$Page.setSecondary(Sort.java:642)
        at uk.me.parabola.imgfmt.app.srt.Sort.finish(Sort.java:161)
        at
uk.me.parabola.mkgmap.srt.SrtTextReader.read(SrtTextReader.java:182)
        at
uk.me.parabola.mkgmap.srt.SrtTextReader.<init>(SrtTextReader.java:117)
        at
uk.me.parabola.mkgmap.srt.SrtTextReader.<init>(SrtTextReader.java:106)
        at
uk.me.parabola.mkgmap.srt.SrtTextReader.sortForCodepage(SrtTextReader.j
ava:141)
        at uk.me.parabola.mkgmap.main.Main.getSort(Main.java:764)
        at
uk.me.parabola.mkgmap.main.Main.processFilename(Main.java:283)
        at
uk.me.parabola.mkgmap.CommandArgsReader$Filename.processArg(CommandArgs
Reader.java:272)
        at
uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java
:125)
        at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
        at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

this is corresponds to:
                                        } else {

                                                p.setSecondary(i,
p.getSecondary(i) + maxSecondary);
                                     
   }

where this increase in the secondary value is governed by some bits in the pages.flags and the flags testing was changed in r3968, along with the introduction of the original assert that was failing (SRTFile.java:117) and quite a few other changes in srt.

@gerd, I'm afraid I don't understand the srt logic so can you have a
look at it

Regards
Ticker

On Sat, 2019-04-06 at 08:16 +0000, Gerd Petermann wrote:
Hi,
maybe a regression from r4167?
http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4167

Gerd

________________________________________
Von: mkgmap-dev [hidden email] im Auftrag
von osm@pinns [hidden email]
Gesendet: Samstag, 6. April 2019 10:01
An: [hidden email]
Betreff: Re: [mkgmap-dev] Building map with Hebrew characters

Have done some checking using older versions

It works on 4143 but not on 4179

I'm afraid I don't have anything in between

r

Nick

On 05/04/2019 21:25, Carlos Dávila wrote:
I'm trying to build a map of Israel with Hebrew characters, but
using
--code-page=1255 a get the following assertion:

at
uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:117
)
        at
uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt4Chars(SRTFile.java:9
9)
        at
uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:63)
        at
uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117
)
        at
uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605)
        at
uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.
java:128)
        at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
        at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

With --unicode I also get an assertion:
Exception in thread "main" java.lang.AssertionError: -125
        at
uk.me.parabola.imgfmt.app.BufferedImgFileWriter.put1u(BufferedImgFi
leWriter.java:154)
        at
uk.me.parabola.imgfmt.app.SectionWriter.put1u(SectionWriter.java:78
)
        at
uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:113
)
        at
uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt8(SRTFile.java:166)
        at
uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:68)
        at
uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117
)
        at
uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605)
        at
uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.
java:128)
        at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
        at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

Probably I'm missing something obvious, but I always used
--code-page=1251 and have no idea what it may be

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Building map with Hebrew characters

nwillink

Just tested 4136 with

--code-page=1255

and it fails  :(

Again, fine with unicode :)

On 08/04/2019 12:03, osm@pinns wrote:

Hi Ticker

I just used unicode

r

Nick

On 08/04/2019 11:38, Ticker Berkin wrote:
Hi Gerd and Nick

@nick, when you say it works on 4143 but not on 4179, did you test code
-page=1255 on 4143, or just unicode?

It looks like there are 2 problems.

The --unicode assertion error is a 'byte' to 'int' sign-extension
problem, introduced in r4167 or r4168 and I've attached a patch that
fixes this and also removes some other confusing/obscuring type
representation changes in this area of the code.

The patch also has 4 commented out (because not appropriate for -
-unicode) asserts that seem to point to the Hebrew/1255 problem being
in imgfmt/app/srt/Sort.java:finish():

$ java -ea -jar /mkgmap/mkgmap.jar --code-page=1255 --gmapsupp test
-map:all-elements

Time started: Mon Apr 08 09:30:06 BST 2019
Exception in thread "main" java.lang.AssertionError: 16
        at
uk.me.parabola.imgfmt.app.srt.Sort$Page.setSecondary(Sort.java:642)
        at uk.me.parabola.imgfmt.app.srt.Sort.finish(Sort.java:161)
        at
uk.me.parabola.mkgmap.srt.SrtTextReader.read(SrtTextReader.java:182)
        at
uk.me.parabola.mkgmap.srt.SrtTextReader.<init>(SrtTextReader.java:117)
        at
uk.me.parabola.mkgmap.srt.SrtTextReader.<init>(SrtTextReader.java:106)
        at
uk.me.parabola.mkgmap.srt.SrtTextReader.sortForCodepage(SrtTextReader.j
ava:141)
        at uk.me.parabola.mkgmap.main.Main.getSort(Main.java:764)
        at
uk.me.parabola.mkgmap.main.Main.processFilename(Main.java:283)
        at
uk.me.parabola.mkgmap.CommandArgsReader$Filename.processArg(CommandArgs
Reader.java:272)
        at
uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java
:125)
        at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
        at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

this is corresponds to:
                                        } else {

                                                p.setSecondary(i,
p.getSecondary(i) + maxSecondary);
                                     
   }

where this increase in the secondary value is governed by some bits in the pages.flags and the flags testing was changed in r3968, along with the introduction of the original assert that was failing (SRTFile.java:117) and quite a few other changes in srt.

@gerd, I'm afraid I don't understand the srt logic so can you have a
look at it

Regards
Ticker

On Sat, 2019-04-06 at 08:16 +0000, Gerd Petermann wrote:
Hi,
maybe a regression from r4167?
http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4167

Gerd

________________________________________
Von: mkgmap-dev [hidden email] im Auftrag
von osm@pinns [hidden email]
Gesendet: Samstag, 6. April 2019 10:01
An: [hidden email]
Betreff: Re: [mkgmap-dev] Building map with Hebrew characters

Have done some checking using older versions

It works on 4143 but not on 4179

I'm afraid I don't have anything in between

r

Nick

On 05/04/2019 21:25, Carlos Dávila wrote:
I'm trying to build a map of Israel with Hebrew characters, but
using
--code-page=1255 a get the following assertion:

at
uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:117
)
        at
uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt4Chars(SRTFile.java:9
9)
        at
uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:63)
        at
uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117
)
        at
uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605)
        at
uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.
java:128)
        at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
        at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

With --unicode I also get an assertion:
Exception in thread "main" java.lang.AssertionError: -125
        at
uk.me.parabola.imgfmt.app.BufferedImgFileWriter.put1u(BufferedImgFi
leWriter.java:154)
        at
uk.me.parabola.imgfmt.app.SectionWriter.put1u(SectionWriter.java:78
)
        at
uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:113
)
        at
uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt8(SRTFile.java:166)
        at
uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:68)
        at
uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117
)
        at
uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605)
        at
uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.
java:128)
        at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
        at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

Probably I'm missing something obvious, but I always used
--code-page=1251 and have no idea what it may be

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Building map with Hebrew characters

Steve Ratcliffe
In reply to this post by Gerd Petermann
Hi

I'll be able to take a look later this evening. I think it is just all sign extension problems.

Steve

On 8 April 2019 11:50:58 BST, Gerd Petermann <[hidden email]> wrote:
Hi Ticker,

sorry, can't help with this part of the source.

@Steve: Please help

Gerd
Von: mkgmap-dev <[hidden email]> im Auftrag von Ticker Berkin <[hidden email]>
Gesendet: Montag, 8. April 2019 12:38
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] Building map with Hebrew characters

Hi Gerd and Nick

@nick, when you say it works on 4143 but not on 4179, did you test code
-page=1255 on 4143, or just unicode?

It looks like there are 2 problems.

The --unicode assertion error is a 'byte' to 'int' sign-extension
problem, introduced in r4167 or r4168 and I've attached a patch that
fixes this and also removes some other confusing/obscuring type
representation changes in this area of the code.

The patch also has 4 commented out (because not appropriate for -
-unicode) asserts that seem to point to the Hebrew/1255 problem being
in imgfmt/app/srt/Sort.java:finish():

$ java -ea -jar /mkgmap/mkgmap.jar --code-page=1255 --gmapsupp test
-map:all-elements

Time started: Mon Apr 08 09:30:06 BST 2019
Exception in thread "main" java.lang.AssertionError: 16
at
uk.me.parabola.imgfmt.app.srt.Sort$Page.setSecondary(Sort.java:642)
at uk.me.parabola.imgfmt.app.srt.Sort.finish(Sort.java:161)
at
uk.me.parabola.mkgmap.srt.SrtTextReader.read(SrtTextReader.java:182)
at
uk.me.parabola.mkgmap.srt.SrtTextReader.<init>(SrtTextReader.java:117)
at
uk.me.parabola.mkgmap.srt.SrtTextReader.<init>(SrtTextReader.java:106)
at
uk.me.parabola.mkgmap.srt.SrtTextReader.sortForCodepage(SrtTextReader.j
ava:141)
at uk.me.parabola.mkgmap.main.Main.getSort(Main.java:764)
at
uk.me.parabola.mkgmap.main.Main.processFilename(Main.java:283)
at
uk.me.parabola.mkgmap.CommandArgsReader$Filename.processArg(CommandArgs
Reader.java:272)
at
uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java
:125)
at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

this is corresponds to:
} else {

p.setSecondary(i,
p.getSecondary(i) + maxSecondary);

}

where this increase in the secondary value is governed by some bits in the pages.flags and the flags testing was changed in r3968, along with the introduction of the original assert that was failing (SRTFile.java:117) and quite a few other changes in srt.

@gerd, I'm afraid I don't understand the srt logic so can you have a
look at it

Regards
Ticker

On Sat, 2019-04-06 at 08:16 +0000, Gerd Petermann wrote:
Hi,
maybe a regression from r4167?
http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4167

Gerd
Von: mkgmap-dev <[hidden email]> im Auftrag
von osm@pinns <[hidden email]>
Gesendet: Samstag, 6. April 2019 10:01
An: [hidden email]
Betreff: Re: [mkgmap-dev] Building map with Hebrew characters

Have done some checking using older versions

It works on 4143 but not on 4179

I'm afraid I don't have anything in between

r

Nick

On 05/04/2019 21:25, Carlos Dávila wrote:
I'm trying to build a map of Israel with Hebrew characters, but
using
--code-page=1255 a get the following assertion:

at
uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:117
)
at
uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt4Chars(SRTFile.java:9
9)
at
uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:63)
at
uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117
)
at
uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605)
at
uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.
java:128)
at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

With --unicode I also get an assertion:
Exception in thread "main" java.lang.AssertionError: -125
at
uk.me.parabola.imgfmt.app.BufferedImgFileWriter.put1u(BufferedImgFi
leWriter.java:154)
at
uk.me.parabola.imgfmt.app.SectionWriter.put1u(SectionWriter.java:78
)
at
uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:113
)
at
uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt8(SRTFile.java:166)
at
uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:68)
at
uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117
)
at
uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605)
at
uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.
java:128)
at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)

Probably I'm missing something obvious, but I always used
--code-page=1251 and have no idea what it may be
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Building map with Hebrew characters

Steve Ratcliffe
In reply to this post by Ticker Berkin

Hi

There is a problem that is specific to cp1255 in that there is a primary
character with more than 15 secondary variations.

I've split the first line up in cp1255.txt for lack of any better idea
of what to do.

The attached patch fixes both 1255 and unicode.

Steve

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

cp.patch (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Building map with Hebrew characters

Ticker Berkin
Hi Gerd

Attached is a patch that takes Steve's changes for Hebrew and a merges
his other changes with my patch from 8th Apr that removes a lot of
pointless swapping between byte/char and int and also fixes the sign
-extension problem.

Regards
Ticker

On Mon, 2019-04-08 at 21:22 +0100, Steve Ratcliffe wrote:

> Hi
>
> There is a problem that is specific to cp1255 in that there is a
> primary
> character with more than 15 secondary variations.
>
> I've split the first line up in cp1255.txt for lack of any better
> idea
> of what to do.
>
> The attached patch fixes both 1255 and unicode.
>
> Steve
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

charSet2.patch (11K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Building map with Hebrew characters

Gerd Petermann
Hi,

@Steve: I see nothing obvious wrong in this patch but I have no idea how to test it. Please commit if you like the changes.

Gerd



________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von Ticker Berkin <[hidden email]>
Gesendet: Montag, 15. April 2019 17:49
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] Building map with Hebrew characters

Hi Gerd

Attached is a patch that takes Steve's changes for Hebrew and a merges
his other changes with my patch from 8th Apr that removes a lot of
pointless swapping between byte/char and int and also fixes the sign
-extension problem.

Regards
Ticker

On Mon, 2019-04-08 at 21:22 +0100, Steve Ratcliffe wrote:

> Hi
>
> There is a problem that is specific to cp1255 in that there is a
> primary
> character with more than 15 secondary variations.
>
> I've split the first line up in cp1255.txt for lack of any better
> idea
> of what to do.
>
> The attached patch fixes both 1255 and unicode.
>
> Steve
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev