Sharing another update from HOT regarding tool development. Last week we released a new version of a data quality monitoring tool called MapCampaigner and want to share it more widely here. MapCampaigner is a tool for tracking and evaluating data quality -- in the form of completeness of OSM tags -- across an area: https://campaigns.hotosm.org
To note a couple things as you look at the tool further:
* Anyone with an OSM username and create a campaign. We do limit the size of the campaign because behind the scenes we're using an Overpass and are limited by capacity to handle country-sized campaigns.
* Data updates on an hourly basis only when the campaign is in an active state -- meaning that the end date is marked in the future.
* We'll be adding back the team tracking functionality in the future. We wanted to get the updates live for people to use as quick as possible. Expect to see some functionality added back over the next couple weeks as we continue to develop.
If you have some feedback or are interested in helping support the development, we'd be glad to have you join the conversation and contribute. This month's HOT Tech Working Group meeting (Oct 2 @ 15:00 UTC) will be demo'ing MapCampaigner, reviewing the progress, and discussing what's next.
Re: [OSM-talk] Tool update from HOT: MapCampaigner
On 01.10.2018 03:28, Nate Smith wrote:
> Last week we
> released a new version of a data quality monitoring tool
I would like to recommend that you don't use the term "quality
monitoring tool" for this since you're measuring quantity not quality.
At best, I'd call it a tool that monitors "richness" or "completeness".
Simply counting how many features there are and how many of a
pre-defined list of tags each one has shouldn't be called "quality
monitoring", because there will be situations where the OpenStreetMap
community requests of project managers (who your web site claims to be
targeted at) that they implement some form of quality assurance; calling
your statistics tool a "quality monitoring" tool runs the risk of making
these people believe that quality requirements can be fulfilled by
ensuring that enough tags are set, which is definitely not what the
wider community would regard as a suitable quality assurance for a
humanitarian data entry project.
Re: [OSM-talk] Tool update from HOT: MapCampaigner
On Mon, October 1, 2018 7:57 am, Frederik Ramm wrote:
> On 01.10.2018 03:28, Nate Smith wrote:
>> Last week we released a new version
>> of a data quality monitoring tool
> I would like to recommend that you don't use the term "quality
> monitoring tool" for this since you're measuring quantity not quality.
> At best, I'd call it a tool that monitors "richness" or "completeness".
And let's underline that Frederik's remark is not semantic nitpicking in a
vacuum: in the context of HOT's campaigns the quality vs. quantity issue
has quite a track record.
Integrating within the task manager the tracking of the quantity of Osmose
defects would go some way towards addressing the monitoring of actual
quality. Tying a selection of Osmose errors categories to the "validation"
step would help make it an actual validation rather than a formal
rubberstamp - doesn't even need to be mandatory, it just needs to be made
visible. Actual quality assurance tools are just there within arm's
Thanks for the comment and questions Frederik. Agreed that quality and quantity isn’t exactly the same. And I hope that the UI in the tool hasn’t communicated that this is just about statistics and quantity - it is more than just a feature counter.
We certainly could say it is a richness monitor, but the goal and purpose is more than that. I’d like to propose that during active field mapping quality in OSM is about the type of tags, spelling or casing errors, and whether you’ve created or updated data according to what you set out to map. This is the feedback we’ve heard from field mapping work. A major purpose of the tool is for tracking tag completeness and quality within a time-bound mapping effort. Tracking quantity is only one aspect to show whats been collected across an area. For many OSM field mapping efforts, quality is a mixture of ensuring that an area is fully covered by visiting on the ground, plus collecting the relevant and useful OSM tags — and adding some spell checking and proper casing checks. That’s why I’m referring to this as a quality monitoring tool - I think it can be used to help improve quality during an active field mapping effort.
This also isn’t focused on producing one-time quality reports for an area and so we’re not limiting the definition of quality only on the humanitarian data model. The goal was to make this flexible and useful to look at any type of tagging schema and track what’s being collected across an area within a defined timeframe. You can set and define what you think is completeness when you’re mapping. The pre-defined list is only a guide for getting started quick. You can track custom individual tags or create a complex custom data model based on what you want to track.
If you have some ideas on additional quality metrics we could add, that would be great as this tool is just getting started. I would definitely like to include more ways we can think about what quality data means during field mapping.
On October 1, 2018 at 12:59:04 PM, Frederik Ramm ([hidden email]) wrote:
On 01.10.2018 03:28, Nate Smith wrote: > Last week we > released a new version of a data quality monitoring tool
I would like to recommend that you don't use the term "quality monitoring tool" for this since you're measuring quantity not quality. At best, I'd call it a tool that monitors "richness" or "completeness".
Simply counting how many features there are and how many of a pre-defined list of tags each one has shouldn't be called "quality monitoring", because there will be situations where the OpenStreetMap community requests of project managers (who your web site claims to be targeted at) that they implement some form of quality assurance; calling your statistics tool a "quality monitoring" tool runs the risk of making these people believe that quality requirements can be fulfilled by ensuring that enough tags are set, which is definitely not what the wider community would regard as a suitable quality assurance for a humanitarian data entry project.
Re: [OSM-dev] [OSM-talk] Tool update from HOT: MapCampaigner
On Mon, Oct 1, 2018 at 3:24 AM Jean-Marc Liotier <[hidden email]> wrote:
On Mon, October 1, 2018 9:56 am, Nate Smith wrote:
If one needs number to report back to donors, then integrate this sort of
thing with the task manager - and explain to donors how erroneous data is
sharply negative value, that an error corrected is even better value than
a new datum and that for their money to have actually usable impact it
needs some share to be allocated to quality control.
This is nothing more than an uneducated attack on my favorite part of the US tax code, 501.c3. This is also nothing more than an attack on a group trying to build the OSM community. All OSM mappers have to go through a the process of being new and growing to be a better mapper. You sound like you expect a baby out of the womb to take that perfect first step without falling down. Yet walking comes some many months later after a body of knowledge is gained by the infant. In my impression of HOT, they are taking people passionate about the human condition and helping that person find an outlet for that passion. That first step might not be the best. The first step for any mapper may not be the best. There is no need to attack HOT based on what a new mapper goes through. Even pub style meetups have to assist mappers through the ropes of OSM education.
This time of year I start planning my final tax bill. I read the reports that you talk about to decide where I will spend my tax deductions and tax credits. I am more worried about overhead for each dollar spent and the minutia that you address and think would make a difference. You see as a you teenager we would go around the day before Halloween with a container asking for pennies for UNICEF. Back then that was the only game in town for charitable work that I knew of. Later I found that only one penny of each dollar that I raised actually when to meet someone's need. I am guessing that 99 cents went to an administrator's pay check. There was no clear accounting. I found a similar issue with United Way. I pick 501.c3 charities based on administrative overhead and how they are trying to break cycles of the working poor's poverty or some other cause. That's right we do not use a Value Added Tax, VAT. After my Adjusted Gross Income, AGI, has been determined, my government allows me to reduce my tax burden by deductions and credits. I have local charities that fill my poverty cycle breakers check box. I also look at each agency's overhead. HOT falls in the same 5% to 15% overhead range that I prefer. As I recall, HOT's admin overhead was 13% for last year. I am guessing that is because of their expensive Washington D.C. address.
Now you need to understand the priority of giving. The best dollar spent is one where I receive a credit for each tax dollar owed. A credit reduces my final tax dollar for dollar. These are mostly state tax credits. Then I move on to deductions. Deductions only reduce my tax partially at so many cents on the dollar. Those deductions are still worthy causes. They are still better causes than letting my federal government blow money on some misguided effort. Included on my short list each year are HOT and the US chapter of OSM. However, by the time I get to this part of my list, I am out of budget for the year. The OSMF is at the very bottom of my list. Money spent there would not be of benefit to my taxes i.e. no 501.c3 designation. Now you can yammer on and on about how the OSM foundation should spend those OSM related dollars but that control does not function in the way my tax system works.
See now that you have a small understanding of how my tax system works verses your VAT, then you can see how I view your remarks as nothing more than a condescending perfectionist attack that expects every mapper to be perfect and full of all the knowledge required to make their first change set contribution. This does nothing more than to destroy the community. We have plenty of QA tools. What is lacking is a community willing to gently grow new mappers to long term mappers.
Re: [OSM-dev] [OSM-talk] Tool update from HOT: MapCampaigner
On Mon, October 8, 2018 1:02 am, Harry Wood wrote:
>> Jean-Marc Liotier <[hidden email]> wrote:
>> If one needs number to report back to donors, then integrate this
>> sort of thing with the task manager - and explain to donors how
>> erroneous data is sharply negative value
> I think this little "number to report back to donors" comment does betray
> a belief I've heard expressed quite often in the wider OSM community, that
> humanitarian mappers are allowing OpenStreetMap to be co-opted by large
> aid organisations [..]
In West Africa (the region where I have direct Openstreetmap experience
and personal engagement with individuals), most organized mapping projects
(rarely the largest contributions in terms of volume of data, but the most
productive ones in terms of enrolling new contributors and weaving
community connections) have at least some financial backing from
organizations, not necessarily large ones. The training cadre spans the
whole spectrum from pure volunteers to professionalized - and those are
accountable to their backers, who expect a bit of reporting, which
includes some measure of quantification. There is nothing wrong with that
perfectly normal requirement but, as any manager knows, the temptation is
strong to fulfil it using the easily available metrics.
Balancing the key performance indicators mix to better align objectives
with Openstreetmap's goals looks like a workable avenue of progress to me
- to everyone's benefit, but that requires conviction about the
complementary merits of quality vs. quantity and consensus about relevant
metrics. Completeness measurements such as MapCampaigner's most certainly
have a role to play, but my perception from wading in West African
changesets is that not producing erroneous data is a higher priority. That
conclusion may of course not apply to other regions I have no experience