Standardized feature/tag database idea & proposal

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Standardized feature/tag database idea & proposal

Seth Deegan
Idea
Has anyone ever thought of creating an official database that stores all of the approved and in-use tags/features in OSM? 

Reasoning
This could allow the editors (iD and JOSM, StreetComplete, GoMap!) and data consumers (osm-carto,. Mapbox etc.), to easily stay up-to-date with new features, without requiring their developers to browse the Wiki etc.

Examples
Both iD and JOSM have their own preset file/repo and are independent of each other. Creating a database that could be used as a dependency of both that would store these feature presets and their fields means:
 - Both are up-to-date and in-sync
 - Adding new presets could be done automatically by retrieving and parsing data from the DB. 

Creating applications that use OSM data can be hard and time-lengthened by requiring developers to browse the Wiki to find all of the features and their keys and values. Having a database that they could easily get the keys they want, their values, etc. would significantly allow greater OSM developer potential.

Specifics
The specifics as to how this database would be arranged such as to where presets/fields/tags/features go has not been thought of yet. I just wanted to ask if this has ever been proposed before. If someone would like me to make a DB layout to help them better understand what I'm proposing, I'd be happy to do so. 

My pre-DB construction proposal 
Before any type of database is made, one would would construct a dedicated page structure on openstreetmap.org that would display and be in-sync with all of the Features from the DB in a format similar to the Wiki, and then remove all of the features from the Wiki altogether.

Why? If you see in Compiling and distributing the DB below, a Wiki bot is going to be required to get all of the pages for all of the already-standard features. Most of the features' pages do have a similar structure as to how they are arranged (template that shows what elements they use, Keys' values and their descriptions are stored in a table, etc.), however these pages would be impossible to parse thoroughly with a computer and are going to require team of humans to get all of their data. 

New features that are proposed and approved after the database would be created would require them to be hand-added. Making a standard way to propose and approve new features on a new part of the website with formatting constraints and database-syncing abilities that MediaWiki cannot offer, would mean that the DB would never have to be physically touched again and ensure greater long-term DB management efficiency.

So why basically delete one of the primary functions of the Wiki and create a new system on openstreetmap.org?
I recently have got into the OSM community head-on in the past two months. I have never really contributed to Wikipedia or any other site that uses the MediaWiki website format so learning about how to contribute and get around the OSM Wiki took time (learning about the proposal process, Templates, talk pages, formatting, the list goes on and on...). It also made me realize that this could discourage new users from ever looking into the depths of OSM or even finding the Wiki at all. The WikiMedia interface is not the prettiest either. It can take time for new users to explore and find what they are looking for.
(Also, this would mean moving the proposal process over to osm.org too)

Therefore, I think creating a easily accessible, pretty, and easily-contributable interface on osm.org would strengthen the OSM community significantly. For example, a heading called "Features" could be added to the left "GPS traces". It would greet the user with the primary features of OSM (kind of like the "Map features" on the Wiki already does) and Feature pages would have a standard format that is consistent throughout the website (unlike Wiki pages where tables for values can be different, proposals can be different, etc.). Pages could also be "locked", or require a proposal before ever changing any of the contents of their page/feature. This would ensure the DB is secure and uniform with the community's agreement on Features (the database is directly synced and edited through changes on the website) and no "random edits' by users like on the current Wiki would have to tracked (since anyone can directly edit). There are other possible benefits that you could probably think of.

Also, other pages on the Wiki would not be deleted. There are plenty of great pages on it that have nothing to do with the DB and work well in the open environment the Wiki has to offer.

Compiling and distributing the DB
  1. One would probably use a Wiki bot like https://en.wikipedia.org/wiki/Wikipedia:AutoWikiBrowser to get all of the features with Categories "approved" and "in-use" (and "depreciated" as well just to let possible future editors know what to get rid of) and add their Keys and Values, descriptions, what elements they are allowed on (nodes, ways, areas, relations), etc. to the DB.
  2. A team would have to go through all of their Key Values that don't have Wiki pages and add them to the DB along with their descriptions, etc.
  3. A team would compare the iD and JOSM present repo and xml file and create a "standard" matching list.
  4. When the DB is finished, iD and JOSM would use it as a dependency and an announcement would be made to data consumers that it is available for use.
Feasibility
This would probably be a huge undertaking and require a funding grant. A plan and dedicated team would be required to ensure all of the tasks are complete and put in-place. I personally think the benefits outweigh the costs, specifically from a developer point-of view. Also, I am not an OSMF member so I'm not sure how much say I would have in making this possible.

Questions? 
Please reply. I like big OSM ideas and am not afraid to get shut-down as I have before with previous ideas. I am clearly a newer contributor, but I hope my ideas and work can foster progress for OSM.

--
Thanks,
Seth Deegan (lectrician1)

_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Standardized feature/tag database idea & proposal

Developer Discussion mailing list



Nov 15, 2020, 19:14 by [hidden email]:
Idea
Has anyone ever thought of creating an official database that stores all of the approved and in-use tags/features in OSM? 
Yes.

Depending on what one means by "database" OSM Wiki can be considered as one.

Wikidata copy ( https://wiki.openstreetmap.org/wiki/Data_items ) is definitely one.
How your proposal differs from data items?

Reasoning
This could allow the editors (iD and JOSM, StreetComplete, GoMap!) and data consumers (osm-carto,. Mapbox etc.), to easily stay up-to-date with new features, without requiring their developers to browse the Wiki etc.
I see no benefit whatsoever of data items (from perspective of someone involved in making
StreetComplete and was active in osm-carto development).

Browsing Wiki, Taginfo, reviewing how tags are actually used and so on would be definitely needed.

Examples
Both iD and JOSM have their own preset file/repo and are independent of each other. Creating a database that could be used as a dependency of both that would store these feature presets and their fields means:
 - Both are up-to-date and in-sync
 - Adding new presets could be done automatically by retrieving and parsing data from the DB. 
There is a good reason why presets are created manually and not pulled from unreviewed
dataset (note that JOSM and iD have separate presets despite that pulling from another preset
would be technically possinle)
Creating applications that use OSM data can be hard and time-lengthened by requiring developers to browse the Wiki to find all of the features and their keys and values. Having a database that they could easily get the keys they want, their values, etc. would significantly allow greater OSM developer potential.
I am unsure what would be difference.
Specifics
The specifics as to how this database would be arranged such as to where presets/fields/tags/features go has not been thought of yet. I just wanted to ask if this has ever been proposed before. If someone would like me to make a DB layout to help them better understand what I'm proposing, I'd be happy to do so. 

My pre-DB construction proposal 

Before any type of database is made, one would would construct a dedicated page structure on openstreetmap.org that would display and be in-sync with all of the Features from the DB in a format similar to the Wiki, and then remove all of the features from the Wiki altogether.

Why? If you see in Compiling and distributing the DB below, a Wiki bot is going to be required to get all of the pages for all of the already-standard features. Most of the features' pages do have a similar structure as to how they are arranged (template that shows what elements they use, Keys' values and their descriptions are stored in a table, etc.), however these pages would be impossible to parse thoroughly with a computer and are going to require team of humans to get all of their data. 

New features that are proposed and approved after the database would be created would require them to be hand-added. Making a standard way to propose and approve new features on a new part of the website with formatting constraints and database-syncing abilities that MediaWiki cannot offer, would mean that the DB would never have to be physically touched again and ensure greater long-term DB management efficiency.

So why basically delete one of the primary functions of the Wiki and create a new system on openstreetmap.org?
I recently have got into the OSM community head-on in the past two months. I have never really contributed to Wikipedia or any other site that uses the MediaWiki website format so learning about how to contribute and get around the OSM Wiki took time (learning about the proposal process, Templates, talk pages, formatting, the list goes on and on...). It also made me realize that this could discourage new users from ever looking into the depths of OSM or even finding the Wiki at all. The WikiMedia interface is not the prettiest either. It can take time for new users to explore and find what they are looking for.
(Also, this would mean moving the proposal process over to osm.org too)

Therefore, I think creating a easily accessible, pretty, and easily-contributable interface on osm.org would strengthen the OSM community significantly. For example, a heading called "Features" could be added to the left "GPS traces". It would greet the user with the primary features of OSM (kind of like the "Map features" on the Wiki already does) and Feature pages would have a standard format that is consistent throughout the website (unlike Wiki pages where tables for values can be different, proposals can be different, etc.). Pages could also be "locked", or require a proposal before ever changing any of the contents of their page/feature. This would ensure the DB is secure and uniform with the community's agreement on Features (the database is directly synced and edited through changes on the website) and no "random edits' by users like on the current Wiki would have to tracked (since anyone can directly edit). There are other possible benefits that you could probably think of.

Also, other pages on the Wiki would not be deleted. There are plenty of great pages on it that have nothing to do with the DB and work well in the open environment the Wiki has to offer.

Compiling and distributing the DB
  1. One would probably use a Wiki bot like https://en.wikipedia.org/wiki/Wikipedia:AutoWikiBrowser to get all of the features with Categories "approved" and "in-use" (and "depreciated" as well just to let possible future editors know what to get rid of) and add their Keys and Values, descriptions, what elements they are allowed on (nodes, ways, areas, relations), etc. to the DB.
  2. A team would have to go through all of their Key Values that don't have Wiki pages and add them to the DB along with their descriptions, etc.
  3. A team would compare the iD and JOSM present repo and xml file and create a "standard" matching list.
  4. When the DB is finished, iD and JOSM would use it as a dependency and an announcement would be made to data consumers that it is available for use.
Feasibility
This would probably be a huge undertaking and require a funding grant. A plan and dedicated team would be required to ensure all of the tasks are complete and put in-place. I personally think the benefits outweigh the costs, specifically from a developer point-of view. Also, I am not an OSMF member so I'm not sure how much say I would have in making this possible.

Questions? 
Please reply. I like big OSM ideas and am not afraid to get shut-down as I have before with previous ideas. I am clearly a newer contributor, but I hope my ideas and work can foster progress for OSM.

--
Thanks,
Seth Deegan (lectrician1)


_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Standardized feature/tag database idea & proposal

Frederik Ramm
In reply to this post by Seth Deegan
Seth,

On 11/15/20 19:14, Seth Deegan wrote:
> Has anyone ever thought of creating an official database that stores all
> of the approved and in-use tags/features in OSM?

Yes, of course, this idea has been around in a variety of flavours. I
remember a conference talk by David Earl in 2010
(https://wiki.openstreetmap.org/wiki/SotM_2010_session:_Tag_Central:_a_Schema_for_OSM)
though even then it was not revolutionary.

I think that going from a wiki to a plain database doesn't actually
solve many problems. Many things that are difficult for people to deal
with - like tag definitions leaving room for interpretation, different
editors having different presets, different language versions describing
tags differently, a lot of vagueness about the proposal process and its
importance, etc.etc. - would not be solved by your idea. You'd only
replace the backend but that would not automatically fix the messy bits
(and for some of the messy bits, "fixing" them might also mean breaking
a part of OSM that many cherish).

For example, one point you mention is that with a database, things could
be somehow "locked", but you're completely ignoring the question of who
would get to lock stuff, who would decide which changes are allowed, and
how this relates to translations, what the appeals process would be
etc.etc. - and *those* are the difficult questions, not the question of
whether I could technically have a mechanism that would restrict editing
to certain people.

And of course, as has been pointed out, the wiki stuff can already be
accessed in a database-like fashion, not only through the data endpoint
but also through the taginfo-wiki SQLite database from
https://taginfo.openstreetmap.org/download.

Bye
Frederik

--
Frederik Ramm  ##  eMail [hidden email]  ##  N49°00'09" E008°23'33"

_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Standardized feature/tag database idea & proposal

SimonPoole
In reply to this post by Seth Deegan

Frederik has already pointed out some of the relevant historic points.

On top of that there is https://github.com/osmlab/editor-presets and I proposed  something similar a couple of years back as a GSOC project.

I would consider the concept moot. On the one hand, while theoretically still possible, the iD preset scheme now has little generalisation and has become very UI specific, requiring to capture a lot more editor specific data than just a couple of years back if you wanted the output to be similar to the original. On the other hand, not just because of the demographics, but also because of the OSMFs involvement, like it or not, the iD presets are the law of the land and anything else is simply irrelevant. That is not a statement about quality (I would maintain that my presets continue to be the best curated set of presets), but about mass.

You should have also given my talk at SOTM 2018 a look https://2018.stateofthemap.org/2018/T061-An_excursion_in_to_the_world_of_OSM_tagging_presets/

In any case both iD and Vespucci use taginfo and associated information to synthesize parts of, or complete presets. Vespucci as part of an explicit search process, iD built in to how their presets work. For a number of reasons the results are at best mixed, but clearly could be improved with some work on taginfo. I would consider this a better way forward than starting something new.

Simon

Am 15.11.2020 um 19:14 schrieb Seth Deegan:
Idea
Has anyone ever thought of creating an official database that stores all of the approved and in-use tags/features in OSM? 

Reasoning
This could allow the editors (iD and JOSM, StreetComplete, GoMap!) and data consumers (osm-carto,. Mapbox etc.), to easily stay up-to-date with new features, without requiring their developers to browse the Wiki etc.

Examples
Both iD and JOSM have their own preset file/repo and are independent of each other. Creating a database that could be used as a dependency of both that would store these feature presets and their fields means:
 - Both are up-to-date and in-sync
 - Adding new presets could be done automatically by retrieving and parsing data from the DB. 

Creating applications that use OSM data can be hard and time-lengthened by requiring developers to browse the Wiki to find all of the features and their keys and values. Having a database that they could easily get the keys they want, their values, etc. would significantly allow greater OSM developer potential.

Specifics
The specifics as to how this database would be arranged such as to where presets/fields/tags/features go has not been thought of yet. I just wanted to ask if this has ever been proposed before. If someone would like me to make a DB layout to help them better understand what I'm proposing, I'd be happy to do so. 

My pre-DB construction proposal 
Before any type of database is made, one would would construct a dedicated page structure on openstreetmap.org that would display and be in-sync with all of the Features from the DB in a format similar to the Wiki, and then remove all of the features from the Wiki altogether.

Why? If you see in Compiling and distributing the DB below, a Wiki bot is going to be required to get all of the pages for all of the already-standard features. Most of the features' pages do have a similar structure as to how they are arranged (template that shows what elements they use, Keys' values and their descriptions are stored in a table, etc.), however these pages would be impossible to parse thoroughly with a computer and are going to require team of humans to get all of their data. 

New features that are proposed and approved after the database would be created would require them to be hand-added. Making a standard way to propose and approve new features on a new part of the website with formatting constraints and database-syncing abilities that MediaWiki cannot offer, would mean that the DB would never have to be physically touched again and ensure greater long-term DB management efficiency.

So why basically delete one of the primary functions of the Wiki and create a new system on openstreetmap.org?
I recently have got into the OSM community head-on in the past two months. I have never really contributed to Wikipedia or any other site that uses the MediaWiki website format so learning about how to contribute and get around the OSM Wiki took time (learning about the proposal process, Templates, talk pages, formatting, the list goes on and on...). It also made me realize that this could discourage new users from ever looking into the depths of OSM or even finding the Wiki at all. The WikiMedia interface is not the prettiest either. It can take time for new users to explore and find what they are looking for.
(Also, this would mean moving the proposal process over to osm.org too)

Therefore, I think creating a easily accessible, pretty, and easily-contributable interface on osm.org would strengthen the OSM community significantly. For example, a heading called "Features" could be added to the left "GPS traces". It would greet the user with the primary features of OSM (kind of like the "Map features" on the Wiki already does) and Feature pages would have a standard format that is consistent throughout the website (unlike Wiki pages where tables for values can be different, proposals can be different, etc.). Pages could also be "locked", or require a proposal before ever changing any of the contents of their page/feature. This would ensure the DB is secure and uniform with the community's agreement on Features (the database is directly synced and edited through changes on the website) and no "random edits' by users like on the current Wiki would have to tracked (since anyone can directly edit). There are other possible benefits that you could probably think of.

Also, other pages on the Wiki would not be deleted. There are plenty of great pages on it that have nothing to do with the DB and work well in the open environment the Wiki has to offer.

Compiling and distributing the DB
  1. One would probably use a Wiki bot like https://en.wikipedia.org/wiki/Wikipedia:AutoWikiBrowser to get all of the features with Categories "approved" and "in-use" (and "depreciated" as well just to let possible future editors know what to get rid of) and add their Keys and Values, descriptions, what elements they are allowed on (nodes, ways, areas, relations), etc. to the DB.
  2. A team would have to go through all of their Key Values that don't have Wiki pages and add them to the DB along with their descriptions, etc.
  3. A team would compare the iD and JOSM present repo and xml file and create a "standard" matching list.
  4. When the DB is finished, iD and JOSM would use it as a dependency and an announcement would be made to data consumers that it is available for use.
Feasibility
This would probably be a huge undertaking and require a funding grant. A plan and dedicated team would be required to ensure all of the tasks are complete and put in-place. I personally think the benefits outweigh the costs, specifically from a developer point-of view. Also, I am not an OSMF member so I'm not sure how much say I would have in making this possible.

Questions? 
Please reply. I like big OSM ideas and am not afraid to get shut-down as I have before with previous ideas. I am clearly a newer contributor, but I hope my ideas and work can foster progress for OSM.

--
Thanks,
Seth Deegan (lectrician1)

_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev

_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev

OpenPGP_0x4721711092E282EA.asc (5K) Download Attachment
OpenPGP_signature (505 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Standardized feature/tag database idea & proposal

Christoph Hormann-2


> Simon Poole <[hidden email]> hat am 16.11.2020 10:21 geschrieben:
>
> [...] On the other hand, not just because of the demographics, but also because of the OSMFs involvement, like it or not, the iD presets are the law of the land and anything else is simply irrelevant.

From my experience with OSM-Carto that is not really accurate.  Whenever iD presets suggest something different from established practice or community consensus the result typically is not the meaning of the tag changing to what iD says.  The result tends to be a huge mess of inconsistent uses.

In other words: iD's power over the development tagging and tag meaning in OSM is primarily either destructive or affirmative.  iD has the power to wreak havoc and destroy a tag or it can affirm and support a trend in already predominantly consistent use.  Same is the case with OSM-Carto.

--
Christoph Hormann
http://www.imagico.de/

_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Standardized feature/tag database idea & proposal

Developer Discussion mailing list
In reply to this post by SimonPoole
Nov 16, 2020, 10:21 by [hidden email]:

the iD presets are the law of the land and anything else is simply irrelevant.

That is so overstating things that it ceases to be an useful statement.

iD presets and
"press button here to `upgrade tags`  without explanation what and why and how actually happens"
are powerful, but it is only a part of what influences tagging.

_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev