OSM tagging validation lib

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

OSM tagging validation lib

François Lacombe-2
Hi

Here is an idea I got regarding tagging validation in editors (iD, JOSM, others).
Subsequently to wiki proposal voting and cleanups, it's currently necessarily to open issues in each editor repository to ask for new tagging validation rules. 

It can sometimes be time consuming to develop those new rules and such a work is done independently by each project maintainer. While each project have its own specific components, background logic is the same.

Would a new lib called like osmtagvalidator or so in charge of doing conform validation to wiki be useful?
It may be shared by any project involved in osm editing and preserve its resources for other valuable developments.

For me, validation doesn't prevent users to use tags they want, but only warn them about possible mistakes.

How would devs and users feel about this?

All the best

François

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: OSM tagging validation lib

James-2
ID is javascript, JOSM is java. So right there I already see a intercompatibility issue

On Dec 24, 2017 6:12 AM, "François Lacombe" <[hidden email]> wrote:
Hi

Here is an idea I got regarding tagging validation in editors (iD, JOSM, others).
Subsequently to wiki proposal voting and cleanups, it's currently necessarily to open issues in each editor repository to ask for new tagging validation rules. 

It can sometimes be time consuming to develop those new rules and such a work is done independently by each project maintainer. While each project have its own specific components, background logic is the same.

Would a new lib called like osmtagvalidator or so in charge of doing conform validation to wiki be useful?
It may be shared by any project involved in osm editing and preserve its resources for other valuable developments.

For me, validation doesn't prevent users to use tags they want, but only warn them about possible mistakes.

How would devs and users feel about this?

All the best

François

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: OSM tagging validation lib

Darafei "Komяpa" Praliaskouski
In reply to this post by François Lacombe-2
Hi,

What language would that library be in, if level0 and iD are in javascript, JOSM and Vespucci are in Java, and many random console tools are in python and perl?

It would be cool to have such library running on osm2pgsql import, fixing and complaining/skipping all the mistaked tag usage it can detect. This can possibly resurrect some objects that are lost because of typos or old tagging schemes.

вс, 24 дек. 2017 г. в 14:12, François Lacombe <[hidden email]>:
Hi

Here is an idea I got regarding tagging validation in editors (iD, JOSM, others).
Subsequently to wiki proposal voting and cleanups, it's currently necessarily to open issues in each editor repository to ask for new tagging validation rules. 

It can sometimes be time consuming to develop those new rules and such a work is done independently by each project maintainer. While each project have its own specific components, background logic is the same.

Would a new lib called like osmtagvalidator or so in charge of doing conform validation to wiki be useful?
It may be shared by any project involved in osm editing and preserve its resources for other valuable developments.

For me, validation doesn't prevent users to use tags they want, but only warn them about possible mistakes.

How would devs and users feel about this?

All the best

François
_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: OSM tagging validation lib

Mateusz Konieczny-2
In reply to this post by François Lacombe-2
conform validation to wiki

Sometimes wiki is wrong and should be changed. 

Note also that authors of different tools have different opinions how and what should be reported as errors.


On 24 Dec 2017 11:12 a.m., "François Lacombe" <[hidden email]> wrote:
Hi

Here is an idea I got regarding tagging validation in editors (iD, JOSM, others).
Subsequently to wiki proposal voting and cleanups, it's currently necessarily to open issues in each editor repository to ask for new tagging validation rules. 

It can sometimes be time consuming to develop those new rules and such a work is done independently by each project maintainer. While each project have its own specific components, background logic is the same.

Would a new lib called like osmtagvalidator or so in charge of doing conform validation to wiki be useful?
It may be shared by any project involved in osm editing and preserve its resources for other valuable developments.

For me, validation doesn't prevent users to use tags they want, but only warn them about possible mistakes.

How would devs and users feel about this?

All the best

François

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk



_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: OSM tagging validation lib

Andy Townsend
In reply to this post by Darafei "Komяpa" Praliaskouski

On 24/12/17 11:49, Darafei "Komяpa" Praliaskouski wrote:
>
>
> It would be cool to have such library running on osm2pgsql import,
> fixing and complaining/skipping all the mistaked tag usage it can
> detect. This can possibly resurrect some objects that are lost because
> of typos or old tagging schemes.
>

That already exists - the lua tag transformation process can do exactly
that.  Here's an example:

https://github.com/SomeoneElseOSM/SomeoneElse-style/blob/master/style.lua#L2056

Obviously whether a particular map style wants to render typing /
tagging errors is a different matter entirely.

Best Regards,

Andy




_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: OSM tagging validation lib

Colin Smale
In reply to this post by James-2

The technical differences between java and JS do not preclude generic thinking. Consider tzdata[1] for example, which does something analogous for time zone data.

The "rules database" can be made portable, in the form of XML or JSON for example. The logic for using these rules can be described in a portable way. Then you add a set of compliance tests, and publish a reference implementation to demonstrate that is is possible to implement it. After that, the logic can be implemented in any language you like, checked against the compliance tests and the bindings published.

Externalising the rules database enables updates and customisations for particular reasons. Depending on the specific use case and the associated non-functionals, validation could possibly be offered as a cloud service (not necessarily by OSM).

 

//colin

[1] https://en.wikipedia.org/wiki/Tz_database

On 2017-12-24 12:18, James wrote:

ID is javascript, JOSM is java. So right there I already see a intercompatibility issue

On Dec 24, 2017 6:12 AM, "François Lacombe" <[hidden email]> wrote:
Hi
 
Here is an idea I got regarding tagging validation in editors (iD, JOSM, others).
Subsequently to wiki proposal voting and cleanups, it's currently necessarily to open issues in each editor repository to ask for new tagging validation rules. 
 
It can sometimes be time consuming to develop those new rules and such a work is done independently by each project maintainer. While each project have its own specific components, background logic is the same.
 
Would a new lib called like osmtagvalidator or so in charge of doing conform validation to wiki be useful?
It may be shared by any project involved in osm editing and preserve its resources for other valuable developments.
 
For me, validation doesn't prevent users to use tags they want, but only warn them about possible mistakes.
 
How would devs and users feel about this?
 
All the best
 
François

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: OSM tagging validation lib

SimonPoole
In reply to this post by Mateusz Konieczny-2

On the one hand lots of the in principle useful information in the wiki is not really easily extractable and on the other hand it is prone to manipulation in more than one way (current fad is to add big warnings about tagging errors what are not).

IMHO addressing the first issue would likely be more helpful and perhaps allow the generation of at least rudimentary presets directly from the wiki (potentially with support from taginfo).

Simon



On 24.12.2017 12:54, Mateusz Konieczny wrote:
conform validation to wiki

Sometimes wiki is wrong and should be changed. 

Note also that authors of different tools have different opinions how and what should be reported as errors.


On 24 Dec 2017 11:12 a.m., "François Lacombe" <[hidden email]> wrote:
Hi

Here is an idea I got regarding tagging validation in editors (iD, JOSM, others).
Subsequently to wiki proposal voting and cleanups, it's currently necessarily to open issues in each editor repository to ask for new tagging validation rules. 

It can sometimes be time consuming to develop those new rules and such a work is done independently by each project maintainer. While each project have its own specific components, background logic is the same.

Would a new lib called like osmtagvalidator or so in charge of doing conform validation to wiki be useful?
It may be shared by any project involved in osm editing and preserve its resources for other valuable developments.

For me, validation doesn't prevent users to use tags they want, but only warn them about possible mistakes.

How would devs and users feel about this?

All the best

François

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk




_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: OSM tagging validation lib

François Lacombe-2
Thanks for answers :)

I said a lib, but indeed the diversity of projects would be too problematic for implementation and rollouts.
JSON, xml, text are fine

As Simon said, taginfo is a great tool to extract data from wiki.
Even if it's not perfect, the wiki is the only reference I know and where data consumers go to get information. Then it's relevent to improve it
It would be hard to put all practicies and rules in wiki keys scorecard templates (from where taginfo get most of its input). Should we complete it in taginfo itself ?

All the best

François 

Le 24 déc. 2017 1:47 PM, "Simon Poole" <[hidden email]> a écrit :

On the one hand lots of the in principle useful information in the wiki is not really easily extractable and on the other hand it is prone to manipulation in more than one way (current fad is to add big warnings about tagging errors what are not).

IMHO addressing the first issue would likely be more helpful and perhaps allow the generation of at least rudimentary presets directly from the wiki (potentially with support from taginfo).

Simon



On 24.12.2017 12:54, Mateusz Konieczny wrote:
conform validation to wiki

Sometimes wiki is wrong and should be changed. 

Note also that authors of different tools have different opinions how and what should be reported as errors.


On 24 Dec 2017 11:12 a.m., "François Lacombe" <[hidden email]> wrote:
Hi

Here is an idea I got regarding tagging validation in editors (iD, JOSM, others).
Subsequently to wiki proposal voting and cleanups, it's currently necessarily to open issues in each editor repository to ask for new tagging validation rules. 

It can sometimes be time consuming to develop those new rules and such a work is done independently by each project maintainer. While each project have its own specific components, background logic is the same.

Would a new lib called like osmtagvalidator or so in charge of doing conform validation to wiki be useful?
It may be shared by any project involved in osm editing and preserve its resources for other valuable developments.

For me, validation doesn't prevent users to use tags they want, but only warn them about possible mistakes.

How would devs and users feel about this?

All the best

François

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk




_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: OSM tagging validation lib

Bryan Housel
In reply to this post by François Lacombe-2
Have you looked at https://github.com/osmlab/osmlint ?
Of all the current validation efforts, that seems like the most promising.  

I’d definitely echo what other people are saying about avoiding the osm wiki if possible. 

It works on vector tiles though, so to stuff it into an editor like iD, we would need to write some kind of pipeline that does:
“current view of stuff in editor” -> "vector tile" -> "osmlint engine" -> “results (geojson)” -> “back to the editor for user to see"

It might work?

Also… This problem of “validating OSM” is really unbounded.  You should know that before you start working on it!  I’m not one to tell people not to work on something but.. It’s really hard!  Tags are just made up all the time by people.

Can a `highway=residential` connect to a `power=line`?  - no!
Can a `highway=service` connect to a `power=substation`  - uhh, I guess!
Can a `highway=??` connect to a `power=thing_i_just_made_up`? - haha!

Bryan




On Dec 24, 2017, at 4:47 AM, François Lacombe <[hidden email]> wrote:

Hi

Here is an idea I got regarding tagging validation in editors (iD, JOSM, others).
Subsequently to wiki proposal voting and cleanups, it's currently necessarily to open issues in each editor repository to ask for new tagging validation rules. 

It can sometimes be time consuming to develop those new rules and such a work is done independently by each project maintainer. While each project have its own specific components, background logic is the same.

Would a new lib called like osmtagvalidator or so in charge of doing conform validation to wiki be useful?
It may be shared by any project involved in osm editing and preserve its resources for other valuable developments.

For me, validation doesn't prevent users to use tags they want, but only warn them about possible mistakes.

How would devs and users feel about this?

All the best

François
_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: OSM tagging validation lib

john whelan-2
But what I suspect could be done is when iD in invoked from a HOT project it could restrict the options on things like highways to those in the wiki  https://wiki.openstreetmap.org/wiki/Highway_Tag_Africa
which would save a lot of living_streets etc. being added in parts of Africa where there aren't any.  Also reduce the numbers of highway=footway rather than path.

We might even get a few more building=yes tags rather than the range we get at the moment.

Thanks John

On 24 December 2017 at 10:45, Bryan Housel <[hidden email]> wrote:
Have you looked at https://github.com/osmlab/osmlint ?
Of all the current validation efforts, that seems like the most promising.  

I’d definitely echo what other people are saying about avoiding the osm wiki if possible. 

It works on vector tiles though, so to stuff it into an editor like iD, we would need to write some kind of pipeline that does:
“current view of stuff in editor” -> "vector tile" -> "osmlint engine" -> “results (geojson)” -> “back to the editor for user to see"

It might work?

Also… This problem of “validating OSM” is really unbounded.  You should know that before you start working on it!  I’m not one to tell people not to work on something but.. It’s really hard!  Tags are just made up all the time by people.

Can a `highway=residential` connect to a `power=line`?  - no!
Can a `highway=service` connect to a `power=substation`  - uhh, I guess!
Can a `highway=??` connect to a `power=thing_i_just_made_up`? - haha!

Bryan




On Dec 24, 2017, at 4:47 AM, François Lacombe <[hidden email]> wrote:

Hi

Here is an idea I got regarding tagging validation in editors (iD, JOSM, others).
Subsequently to wiki proposal voting and cleanups, it's currently necessarily to open issues in each editor repository to ask for new tagging validation rules. 

It can sometimes be time consuming to develop those new rules and such a work is done independently by each project maintainer. While each project have its own specific components, background logic is the same.

Would a new lib called like osmtagvalidator or so in charge of doing conform validation to wiki be useful?
It may be shared by any project involved in osm editing and preserve its resources for other valuable developments.

For me, validation doesn't prevent users to use tags they want, but only warn them about possible mistakes.

How would devs and users feel about this?

All the best

François
_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk



_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: OSM tagging validation lib

François Lacombe-2
In reply to this post by Bryan Housel
Hi Bryan

Le 24 déc. 2017 4:45 PM, "Bryan Housel" <[hidden email]> a écrit :
Have you looked at https://github.com/osmlab/osmlint ?
Of all the current validation efforts, that seems like the most promising.

I didn't know OSMLint and OSM QA tiles before
Very promising indeed for parallel processing
Issue I see it's relations aren't available unfortunately


I’d definitely echo what other people are saying about avoiding the osm wiki if possible.
Can you elaborate please ?
I just don't know elsewhere anyone can find comprehensive and consistent information about tags despite wiki is not always perfect
Wiki got good functionalities to log contributions and revert vandalism too.


It works on vector tiles though, so to stuff it into an editor like iD, we would need to write some kind of pipeline that does:
“current view of stuff in editor” -> "vector tile" -> "osmlint engine" -> “results (geojson)” -> “back to the editor for user to see"

It might work?
It can clearly work :)
Nevertheless it's one usecase out of plenty
Validation systems can be used to do data audit too
That's why focusing on rules formatting is more versatile than writing implementation unlinke what i was originally suggesting



Also… This problem of “validating OSM” is really unbounded.  You should know that before you start working on it!  I’m not one to tell people not to work on something but.. It’s really hard!  Tags are just made up all the time by people.

I agree and it's a different problem
Focusing on rules formalism doesn't assume what rules should be.
Even if tags are made by people, some definitions can be commonly accepted and they can be refined after some discussion. Validation can follow the same peocess also.


Can a `highway=residential` connect to a `power=line`?  - no!
Can a `highway=service` connect to a `power=substation`  - uhh, I guess!
Can a `highway=??` connect to a `power=thing_i_just_made_up`? - haha!
These are rules, not the description we should build to make them understandable by software

All the best

François 

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: OSM tagging validation lib

Bryan Housel
In reply to this post by john whelan-2
What you are describing has nothing to do with validation. This could be done now without any new development just by running a copy of iD with different presets. 

Sent from my iPhone

On Dec 24, 2017, at 12:08 PM, john whelan <[hidden email]> wrote:

But what I suspect could be done is when iD in invoked from a HOT project it could restrict the options on things like highways to those in the wiki  https://wiki.openstreetmap.org/wiki/Highway_Tag_Africa
which would save a lot of living_streets etc. being added in parts of Africa where there aren't any.  Also reduce the numbers of highway=footway rather than path.

We might even get a few more building=yes tags rather than the range we get at the moment.

Thanks John

On 24 December 2017 at 10:45, Bryan Housel <[hidden email]> wrote:
Have you looked at https://github.com/osmlab/osmlint ?
Of all the current validation efforts, that seems like the most promising.  

I’d definitely echo what other people are saying about avoiding the osm wiki if possible. 

It works on vector tiles though, so to stuff it into an editor like iD, we would need to write some kind of pipeline that does:
“current view of stuff in editor” -> "vector tile" -> "osmlint engine" -> “results (geojson)” -> “back to the editor for user to see"

It might work?

Also… This problem of “validating OSM” is really unbounded.  You should know that before you start working on it!  I’m not one to tell people not to work on something but.. It’s really hard!  Tags are just made up all the time by people.

Can a `highway=residential` connect to a `power=line`?  - no!
Can a `highway=service` connect to a `power=substation`  - uhh, I guess!
Can a `highway=??` connect to a `power=thing_i_just_made_up`? - haha!

Bryan




On Dec 24, 2017, at 4:47 AM, François Lacombe <[hidden email]> wrote:

Hi

Here is an idea I got regarding tagging validation in editors (iD, JOSM, others).
Subsequently to wiki proposal voting and cleanups, it's currently necessarily to open issues in each editor repository to ask for new tagging validation rules. 

It can sometimes be time consuming to develop those new rules and such a work is done independently by each project maintainer. While each project have its own specific components, background logic is the same.

Would a new lib called like osmtagvalidator or so in charge of doing conform validation to wiki be useful?
It may be shared by any project involved in osm editing and preserve its resources for other valuable developments.

For me, validation doesn't prevent users to use tags they want, but only warn them about possible mistakes.

How would devs and users feel about this?

All the best

François
_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk



_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: OSM tagging validation lib

john whelan-2
True but it's to do with data quality and avoiding the need to validate.

Thank you for the comment.

Cheerio John

On 24 Dec 2017 1:37 pm, "Bryan Housel" <[hidden email]> wrote:
What you are describing has nothing to do with validation. This could be done now without any new development just by running a copy of iD with different presets. 

Sent from my iPhone

On Dec 24, 2017, at 12:08 PM, john whelan <[hidden email]> wrote:

But what I suspect could be done is when iD in invoked from a HOT project it could restrict the options on things like highways to those in the wiki  https://wiki.openstreetmap.org/wiki/Highway_Tag_Africa
which would save a lot of living_streets etc. being added in parts of Africa where there aren't any.  Also reduce the numbers of highway=footway rather than path.

We might even get a few more building=yes tags rather than the range we get at the moment.

Thanks John

On 24 December 2017 at 10:45, Bryan Housel <[hidden email]> wrote:
Have you looked at https://github.com/osmlab/osmlint ?
Of all the current validation efforts, that seems like the most promising.  

I’d definitely echo what other people are saying about avoiding the osm wiki if possible. 

It works on vector tiles though, so to stuff it into an editor like iD, we would need to write some kind of pipeline that does:
“current view of stuff in editor” -> "vector tile" -> "osmlint engine" -> “results (geojson)” -> “back to the editor for user to see"

It might work?

Also… This problem of “validating OSM” is really unbounded.  You should know that before you start working on it!  I’m not one to tell people not to work on something but.. It’s really hard!  Tags are just made up all the time by people.

Can a `highway=residential` connect to a `power=line`?  - no!
Can a `highway=service` connect to a `power=substation`  - uhh, I guess!
Can a `highway=??` connect to a `power=thing_i_just_made_up`? - haha!

Bryan




On Dec 24, 2017, at 4:47 AM, François Lacombe <[hidden email]> wrote:

Hi

Here is an idea I got regarding tagging validation in editors (iD, JOSM, others).
Subsequently to wiki proposal voting and cleanups, it's currently necessarily to open issues in each editor repository to ask for new tagging validation rules. 

It can sometimes be time consuming to develop those new rules and such a work is done independently by each project maintainer. While each project have its own specific components, background logic is the same.

Would a new lib called like osmtagvalidator or so in charge of doing conform validation to wiki be useful?
It may be shared by any project involved in osm editing and preserve its resources for other valuable developments.

For me, validation doesn't prevent users to use tags they want, but only warn them about possible mistakes.

How would devs and users feel about this?

All the best

François
_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk



_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: OSM tagging validation lib

Andy Townsend
On 24/12/17 18:41, john whelan wrote:
> True but it's to do with data quality and avoiding the need to validate.

I don't think that different presets will "avoid the need to validate"
in any scenario (HOT or otherwise) - the only thing that will do that is
education and training - and having those educated and trained people
come back again to do more mapping.

Best Regards,

Andy


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: OSM tagging validation lib

john whelan-2
The problem with HOT inspired maperthons is half the participants will not return.  So restricting their choices to the most common is helpful.

Typically I see highway=primary between two small settlements, highway=living_steet by mappers mapping from imagery in African countries that do not have any in legal existence.

Whilst there is a mechanism of validation to correct errors it's only very recently that it has been possible to restrict validation to mappers with some experience.  Even today with TM3 I can see tiles have been validated by mappers with very little or even no experience and it is not possible to identify the tiles easily to revalidate them and to be honest not that much HOT mapping gets validated.

Even with mappers who use JOSM and do not map through HOT there are problems.  One cheerful mapper managed to upload over 400 untagged ways with JOSM and they would have been warned when uploading them.

It isn't possible to make things idiot proof new improved idiots arrive all the time but by suggesting the most common tags to new mappers might make it easier for them to select the correct values.

Cheerio John

On 24 Dec 2017 1:51 pm, "Andy Townsend" <[hidden email]> wrote:
On 24/12/17 18:41, john whelan wrote:
True but it's to do with data quality and avoiding the need to validate.

I don't think that different presets will "avoid the need to validate" in any scenario (HOT or otherwise) - the only thing that will do that is education and training - and having those educated and trained people come back again to do more mapping.

Best Regards,

Andy


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: OSM tagging validation lib

Yuri Astrakhan-2
In reply to this post by Colin Smale
Declarative rules are usually not very good. Every tool must understand every type of rule, and must be updated when new rule types are introduced. Plus declarative grammar is either too limiting, or eventually starts looking like a scripting language itself, and we end up building an execution environment in every tool.

I think a better path is executing scripts inside other languages, e.g.
* a JavaScript library ran by Java, Python, C++, ...
* a lib that gets compiled into a webassembly for browser, or connected to other languages via native bindings (less tried path)

The lib would need an API to
* access local data state
* access master OSM DB for additional data
* access other tools like taginfo

Integrating scripting environment may be difficult, but offers far greater benefits of rule consistency and flexibility.


On Sun, Dec 24, 2017 at 7:30 AM, Colin Smale <[hidden email]> wrote:

The technical differences between java and JS do not preclude generic thinking. Consider tzdata[1] for example, which does something analogous for time zone data.

The "rules database" can be made portable, in the form of XML or JSON for example. The logic for using these rules can be described in a portable way. Then you add a set of compliance tests, and publish a reference implementation to demonstrate that is is possible to implement it. After that, the logic can be implemented in any language you like, checked against the compliance tests and the bindings published.

Externalising the rules database enables updates and customisations for particular reasons. Depending on the specific use case and the associated non-functionals, validation could possibly be offered as a cloud service (not necessarily by OSM).

 

//colin

[1] https://en.wikipedia.org/wiki/Tz_database

On 2017-12-24 12:18, James wrote:

ID is javascript, JOSM is java. So right there I already see a intercompatibility issue

On Dec 24, 2017 6:12 AM, "François Lacombe" <[hidden email]> wrote:
Hi
 
Here is an idea I got regarding tagging validation in editors (iD, JOSM, others).
Subsequently to wiki proposal voting and cleanups, it's currently necessarily to open issues in each editor repository to ask for new tagging validation rules. 
 
It can sometimes be time consuming to develop those new rules and such a work is done independently by each project maintainer. While each project have its own specific components, background logic is the same.
 
Would a new lib called like osmtagvalidator or so in charge of doing conform validation to wiki be useful?
It may be shared by any project involved in osm editing and preserve its resources for other valuable developments.
 
For me, validation doesn't prevent users to use tags they want, but only warn them about possible mistakes.
 
How would devs and users feel about this?
 
All the best
 
François

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk



_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: OSM tagging validation lib

Colin Smale

Hi Yuri,

We have to decide what is most important - fidelity to the infinite number of tagging styles out there, or the ability to get a basic set of tagging grammar accepted in as many tools as possible.

Any rules grammar will always have limits of course. If the rules are too complex to represent in a declarative way, that is in itself an indication of the mess we have got ourselves into. If the "unwritten rules" of OSM tagging are too complex to write down, then they need sorting out first! Having a simple base to work from might be a good first step. Automated chaos is still chaos.

I agree that the ultimate rules engine may well end up using e.g. JS as a medium to express some of the subtleties of the rules. Once a JS implementation has been published, then people can translate the JS reference implementation into whatever language they need. But separating the basic rules from the execution engine is nothing more than architectural best practice, and there is no reason that the basic rules should not be portable across runtime environments.

Can you think of any complex patterns which cannot (easily) be expressed in a declarative way?

The real challenge here is not for the coders, but a perennial challenge for the OSM community. How do we get to such a consensus about tagging patterns, that we can actually say "this is correct" and "this is wrong enough to warrant correction" without upsetting a large number of people? As soon as a discussion is about right vs wrong, it degenerates into mudslinging.

 

//colin

On 2017-12-24 20:37, Yuri Astrakhan wrote:

Declarative rules are usually not very good. Every tool must understand every type of rule, and must be updated when new rule types are introduced. Plus declarative grammar is either too limiting, or eventually starts looking like a scripting language itself, and we end up building an execution environment in every tool.
 
 
I think a better path is executing scripts inside other languages, e.g.
 
* a JavaScript library ran by Java, Python, C++, ...
* a lib that gets compiled into a webassembly for browser, or connected to other languages via native bindings (less tried path)
 
The lib would need an API to
* access local data state
* access master OSM DB for additional data
* access other tools like taginfo
 
 
Integrating scripting environment may be difficult, but offers far greater benefits of rule consistency and flexibility.
 

On Sun, Dec 24, 2017 at 7:30 AM, Colin Smale <[hidden email]> wrote:

The technical differences between java and JS do not preclude generic thinking. Consider tzdata[1] for example, which does something analogous for time zone data.

The "rules database" can be made portable, in the form of XML or JSON for example. The logic for using these rules can be described in a portable way. Then you add a set of compliance tests, and publish a reference implementation to demonstrate that is is possible to implement it. After that, the logic can be implemented in any language you like, checked against the compliance tests and the bindings published.

Externalising the rules database enables updates and customisations for particular reasons. Depending on the specific use case and the associated non-functionals, validation could possibly be offered as a cloud service (not necessarily by OSM).

 

//colin

[1] https://en.wikipedia.org/wiki/Tz_database

On 2017-12-24 12:18, James wrote:

ID is javascript, JOSM is java. So right there I already see a intercompatibility issue

On Dec 24, 2017 6:12 AM, "François Lacombe" <[hidden email]> wrote:
Hi
 
Here is an idea I got regarding tagging validation in editors (iD, JOSM, others).
Subsequently to wiki proposal voting and cleanups, it's currently necessarily to open issues in each editor repository to ask for new tagging validation rules. 
 
It can sometimes be time consuming to develop those new rules and such a work is done independently by each project maintainer. While each project have its own specific components, background logic is the same.
 
Would a new lib called like osmtagvalidator or so in charge of doing conform validation to wiki be useful?
It may be shared by any project involved in osm editing and preserve its resources for other valuable developments.
 
For me, validation doesn't prevent users to use tags they want, but only warn them about possible mistakes.
 
How would devs and users feel about this?
 
All the best
 
François

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: OSM tagging validation lib

Yuri Astrakhan-2
Colin, I think we should look at JOSM's validations as a good example of what is needed. For some validations, it extended MapCSS with its own custom quirks, and even added its own unit testing. For other types of validations (e.g. geometries), it seems to fall back to full code.  Having a single language to express any types of validation would simplify maintenance and participation.

I don't think people should translate the JS reference implementation at all. That recreates the problem we have now. Instead, all tools should use one common library directly. The only problem is integration - Java, Python, and C++ can all call JS functions, we just need to ensure it is *reasonably* fast and easy to deploy.

Lastly, I am sure we will want to introduce "slow validations" - when a call to an external service, e.g. OSM db or taginfo is required. Those may be used by some of the tools, e.g. editors right before saving.

Thx!

On Sun, Dec 24, 2017 at 3:03 PM, Colin Smale <[hidden email]> wrote:

Hi Yuri,

We have to decide what is most important - fidelity to the infinite number of tagging styles out there, or the ability to get a basic set of tagging grammar accepted in as many tools as possible.

Any rules grammar will always have limits of course. If the rules are too complex to represent in a declarative way, that is in itself an indication of the mess we have got ourselves into. If the "unwritten rules" of OSM tagging are too complex to write down, then they need sorting out first! Having a simple base to work from might be a good first step. Automated chaos is still chaos.

I agree that the ultimate rules engine may well end up using e.g. JS as a medium to express some of the subtleties of the rules. Once a JS implementation has been published, then people can translate the JS reference implementation into whatever language they need. But separating the basic rules from the execution engine is nothing more than architectural best practice, and there is no reason that the basic rules should not be portable across runtime environments.

Can you think of any complex patterns which cannot (easily) be expressed in a declarative way?

The real challenge here is not for the coders, but a perennial challenge for the OSM community. How do we get to such a consensus about tagging patterns, that we can actually say "this is correct" and "this is wrong enough to warrant correction" without upsetting a large number of people? As soon as a discussion is about right vs wrong, it degenerates into mudslinging.

 

//colin

On 2017-12-24 20:37, Yuri Astrakhan wrote:

Declarative rules are usually not very good. Every tool must understand every type of rule, and must be updated when new rule types are introduced. Plus declarative grammar is either too limiting, or eventually starts looking like a scripting language itself, and we end up building an execution environment in every tool.
 
 
I think a better path is executing scripts inside other languages, e.g.
 
* a JavaScript library ran by Java, Python, C++, ...
* a lib that gets compiled into a webassembly for browser, or connected to other languages via native bindings (less tried path)
 
The lib would need an API to
* access local data state
* access master OSM DB for additional data
* access other tools like taginfo
 
 
Integrating scripting environment may be difficult, but offers far greater benefits of rule consistency and flexibility.
 

On Sun, Dec 24, 2017 at 7:30 AM, Colin Smale <[hidden email]> wrote:

The technical differences between java and JS do not preclude generic thinking. Consider tzdata[1] for example, which does something analogous for time zone data.

The "rules database" can be made portable, in the form of XML or JSON for example. The logic for using these rules can be described in a portable way. Then you add a set of compliance tests, and publish a reference implementation to demonstrate that is is possible to implement it. After that, the logic can be implemented in any language you like, checked against the compliance tests and the bindings published.

Externalising the rules database enables updates and customisations for particular reasons. Depending on the specific use case and the associated non-functionals, validation could possibly be offered as a cloud service (not necessarily by OSM).

 

//colin

[1] https://en.wikipedia.org/wiki/Tz_database

On 2017-12-24 12:18, James wrote:

ID is javascript, JOSM is java. So right there I already see a intercompatibility issue

On Dec 24, 2017 6:12 AM, "François Lacombe" <[hidden email]> wrote:
Hi
 
Here is an idea I got regarding tagging validation in editors (iD, JOSM, others).
Subsequently to wiki proposal voting and cleanups, it's currently necessarily to open issues in each editor repository to ask for new tagging validation rules. 
 
It can sometimes be time consuming to develop those new rules and such a work is done independently by each project maintainer. While each project have its own specific components, background logic is the same.
 
Would a new lib called like osmtagvalidator or so in charge of doing conform validation to wiki be useful?
It may be shared by any project involved in osm editing and preserve its resources for other valuable developments.
 
For me, validation doesn't prevent users to use tags they want, but only warn them about possible mistakes.
 
How would devs and users feel about this?
 
All the best
 
François

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: OSM tagging validation lib

Frederik Ramm
In reply to this post by Colin Smale
Hi,

On 24.12.2017 21:03, Colin Smale wrote:
> The real challenge here is not for the coders, but a perennial challenge
> for the OSM community. How do we get to such a consensus about tagging
> patterns, that we can actually say "this is correct" and "this is wrong
> enough to warrant correction" without upsetting a large number of
> people?

Even upsetting a *small* number of people could be considered a problem.
If you look at this from the much-touted diversity perspective, why
should German mappers who are a majority by numbers, dictate to a small
group of mappers in Peru which tags they should be using? Why should men
who are a majority by numbers, decide which tags women should be using? Etc.

Some decisions are necessary for the project to function, and hence
taking them is important, even if the mechanisms by which they are taken
may be flawed, and suppress minorities.

But we should be careful about applying our necessarily flawed decision
making mechanisms to *too many* aspects of mapping; every time you
narrow down the envelope of "acceptable" tagging, you could be using
your superior numbers power to put someone somewhere at a disadvantage
or deprive them of a voice.

People sometimes say that every decision not taken contributes to a
vacuum that decreases the quality of mapping; but it is equally valid to
say that every decision not taken contributes to the freedom to map what
you want, how you want it.

As always, the truth is somewhere in between.

Bye
Frederik

--
Frederik Ramm  ##  eMail [hidden email]  ##  N49°00'09" E008°23'33"

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: OSM tagging validation lib

SimonPoole
In reply to this post by Yuri Astrakhan-2
We already have validation tools in many forms for post-editing
validation and already know how difficult they are to get "right" (see
OSMOSE complaining about P+R facilities for a trivial example). I don't
quite see why there is a need to create yet another one/system.

On the other hand validation on upload is horrible from a UI and
workflow point of view (which is why JOSMs warnings get ignored so
often). IMHO preventing inadvertent mistakes to start with seems to be
far better and that can be done with consistent presets (which might
have to be localized) and editor support (see what Bryan has just done
with creating area geometries in iD).

Simon

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
12