Client side testing

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Client side testing

Rory O'Donnell

Hi Vincent,

I noticed that JOSM have been commended in the last two Quality reports
[1] [2] for the number and
indeed the quality of bugs you have found and logged against various
early access releases.
Thanks again for your contribution to the quality of our releases!

The big question is how do you do it ?

We would love to hear more details about your test environment, what
tests you run etc. Below is a
series of questions that come to mind, but there may be other details
that you would like to share.

 1. Builds
     1. do you build your own OpenJDK builds, use Oracle's or other EA
        builds ?
 2. Tests
     1. What tests do you run
         1. do you run the JCK ?
         2. open regression tests ?
         3. other tests ?
         4. do you measure code coverage ?
     2. Are the tests Automated or Manual ?
     3. How frequently to do execute the tests
     4. Are the results visible via an external link

Rgds, Rory

[1]
https://wiki.openjdk.java.net/display/quality/Quality+Outreach+report+September+2017
[2]
https://wiki.openjdk.java.net/display/quality/Quality+Outreach+report+March+2017

--
Rgds,Rory O'Donnell
Quality Engineering Manager
Oracle EMEA , Dublin, Ireland

Reply | Threaded
Open this post in threaded view
|

Re: Client side testing

Vincent Privat
Hello Rory,
Thank you :)

Here is how we test various OpenJDK builds:
1.1: we use only EA builds from http://jdk.java.net/
2.1.1: no
2.1.2/2.1.3: we run all unit and integration tests of JOSM
2.1.4: yes, using jacoco
2.2: all tests are automated
2.3: it depends. During OpenJDK9 cycle we did not run a test series for
each weekly build, but rather once every two or three builds, unless we
were waiting for a specific correction from OpenJDK or another tool. At the
end of development, we were not waiting for any correction from OpenJDK but
were eagerly waiting for the updates of ASM and everything that as blocked
by the non-availability of ASM (in particular various Maven plugins)
2.4: Yes, they are visible here:
- JOSM core:
https://josm.openstreetmap.de/jenkins/job/Java-EarlyAccess-JOSM/
- JOSM community plugins:
https://josm.openstreetmap.de/jenkins/job/Java-EarlyAccess-JOSM-Plugins/

I just have initiated JDK10 tests today, and we have 53 failures caused by
Byte Buddy not recognizing Java version 10:

java.lang.IllegalArgumentException: Unknown Java version: 10
        at nl.jqno.equalsverifier.internal.lib.bytebuddy.ClassFileVersion.ofJavaVersion(ClassFileVersion.java:135)


I'm sure Rafael is aware of it. I will see how to update our tools
(equalsverifier here).

Cheers,
Vincent

2017-11-03 14:05 GMT+01:00 Rory O'Donnell <[hidden email]>:

>
> Hi Vincent,
> I noticed that JOSM have been commended in the last two Quality reports
> [1] [2] for the number and
> indeed the quality of bugs you have found and logged against various early
> access releases.
> Thanks again for your contribution to the quality of our releases!
>
> The big question is how do you do it ?
>
> We would love to hear more details about your test environment, what tests
> you run etc. Below is a
> series of questions that come to mind, but there may be other details that
> you would like to share.
>
>    1. Builds
>       1. do you build your own OpenJDK builds, use Oracle's or other EA
>       builds ?
>    2. Tests
>       1. What tests do you run
>       1. do you run the JCK ?
>          2. open regression tests ?
>          3. other tests ?
>          4. do you measure code coverage ?
>          2. Are the tests Automated or Manual ?
>       3. How frequently to do execute the tests
>       4. Are the results visible via an external link
>
> Rgds, Rory
> [1] https://wiki.openjdk.java.net/display/quality/Quality+
> Outreach+report+September+2017
> [2] https://wiki.openjdk.java.net/display/quality/Quality+
> Outreach+report+March+2017
>
> --
> Rgds,Rory O'Donnell
> Quality Engineering Manager
> Oracle EMEA , Dublin, Ireland
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Client side testing

Rory O'Donnell
Hi Vincent,

Thanks for the update, plenty of info for us to review. We may comeback
with follow on questions.

Rgds,Rory
On 03/11/2017 15:33, Vincent Privat wrote:

> Hello Rory,
> Thank you :)
>
> Here is how we test various OpenJDK builds:
> 1.1: we use only EA builds from http://jdk.java.net/
> 2.1.1: no
> 2.1.2/2.1.3: we run all unit and integration tests of JOSM
> 2.1.4: yes, using jacoco
> 2.2: all tests are automated
> 2.3: it depends. During OpenJDK9 cycle we did not run a test series
> for each weekly build, but rather once every two or three builds,
> unless we were waiting for a specific correction from OpenJDK or
> another tool. At the end of development, we were not waiting for any
> correction from OpenJDK but were eagerly waiting for the updates of
> ASM and everything that as blocked by the non-availability of ASM (in
> particular various Maven plugins)
> 2.4: Yes, they are visible here:
> - JOSM core:
> https://josm.openstreetmap.de/jenkins/job/Java-EarlyAccess-JOSM/
> - JOSM community plugins:
> https://josm.openstreetmap.de/jenkins/job/Java-EarlyAccess-JOSM-Plugins/
>
> I just have initiated JDK10 tests today, and we have 53 failures
> caused by Byte Buddy not recognizing Java version 10:
>
> java.lang.IllegalArgumentException: Unknown Java version: 10
> at nl.jqno.equalsverifier.internal.lib.bytebuddy.ClassFileVersion.ofJavaVersion(ClassFileVersion.java:135)
>
> I'm sure Rafael is aware of it. I will see how to update our tools
> (equalsverifier here).
>
> Cheers,
> Vincent
>
> 2017-11-03 14:05 GMT+01:00 Rory O'Donnell <[hidden email]
> <mailto:[hidden email]>>:
>
>
>     Hi Vincent,
>
>     I noticed that JOSM have been commended in the last two Quality
>     reports [1] [2] for the number and
>     indeed the quality of bugs you have found and logged against
>     various early access releases.
>     Thanks again for your contribution to the quality of our releases!
>
>     The big question is how do you do it ?
>
>     We would love to hear more details about your test environment,
>     what tests you run etc. Below is a
>     series of questions that come to mind, but there may be other
>     details that you would like to share.
>
>      1. Builds
>          1. do you build your own OpenJDK builds, use Oracle's or
>             other EA builds ?
>      2. Tests
>          1. What tests do you run
>              1. do you run the JCK ?
>              2. open regression tests ?
>              3. other tests ?
>              4. do you measure code coverage ?
>          2. Are the tests Automated or Manual ?
>          3. How frequently to do execute the tests
>          4. Are the results visible via an external link
>
>     Rgds, Rory
>
>     [1]
>     https://wiki.openjdk.java.net/display/quality/Quality+Outreach+report+September+2017
>     <https://wiki.openjdk.java.net/display/quality/Quality+Outreach+report+September+2017>
>     [2]
>     https://wiki.openjdk.java.net/display/quality/Quality+Outreach+report+March+2017
>     <https://wiki.openjdk.java.net/display/quality/Quality+Outreach+report+March+2017>
>
>     --
>     Rgds,Rory O'Donnell
>     Quality Engineering Manager
>     Oracle EMEA , Dublin, Ireland
>
>

--
Rgds,Rory O'Donnell
Quality Engineering Manager
Oracle EMEA , Dublin, Ireland

Reply | Threaded
Open this post in threaded view
|

Re: Client side testing

Vincent Privat
In reply to this post by Vincent Privat
Sure! Currently coverage is restricted to JOSM classes, for obvious
performance reasons, as we run a full build after each commit.
But I can add a parameter to change the monitored packages in our EA
builds.
Is it OK if I add java, javax, sun and com.sun ? Or do you prefer to
restrict only to packages from java.desktop module?

2017-11-06 19:31 GMT+01:00 Alexandre (Shura) Iline <
[hidden email]>:

> This is great that you are touching the new builds, thank you, Vincent.
>
> I am not familiar with JOSM code, but the app is great! I have played with
> it in the past.
>
> Do you think it would be possible, in addition to what you are doing, to
> collect code coverage for the  JDK code? Specifically for the now existing
> java.desktop module. I do not know much of JaCoCo - we are using JCov, as
> you can imagine - so I do not know how easy it is. If it is easy, can you
> flip the switch?
>
> Shura
> > On Nov 3, 2017, at 8:33 AM, Vincent Privat <[hidden email]>
> wrote:
> >
> > Hello Rory,
> > Thank you :)
> >
> > Here is how we test various OpenJDK builds:
> > 1.1: we use only EA builds from http://jdk.java.net/
> > 2.1.1: no
> > 2.1.2/2.1.3: we run all unit and integration tests of JOSM
> > 2.1.4: yes, using jacoco
> > 2.2: all tests are automated
> > 2.3: it depends. During OpenJDK9 cycle we did not run a test series for
> each weekly build, but rather once every two or three builds, unless we
> were waiting for a specific correction from OpenJDK or another tool. At the
> end of development, we were not waiting for any correction from OpenJDK but
> were eagerly waiting for the updates of ASM and everything that as blocked
> by the non-availability of ASM (in particular various Maven plugins)
> > 2.4: Yes, they are visible here:
> > - JOSM core: https://josm.openstreetmap.de/jenkins/job/Java-EarlyAccess-
> JOSM/
> > - JOSM community plugins: https://josm.openstreetmap.de/
> jenkins/job/Java-EarlyAccess-JOSM-Plugins/
> >
> > I just have initiated JDK10 tests today, and we have 53 failures caused
> by Byte Buddy not recognizing Java version 10:
> >
> > java.lang.IllegalArgumentException: Unknown Java version: 10
> >       at nl.jqno.equalsverifier.internal.lib.bytebuddy.ClassFileVersion.
> ofJavaVersion(ClassFileVersion.java:135)
> >
> >
> > I'm sure Rafael is aware of it. I will see how to update our tools
> (equalsverifier here).
> >
> > Cheers,
> > Vincent
> >
> > 2017-11-03 14:05 GMT+01:00 Rory O'Donnell <[hidden email]>:
> >
> > Hi Vincent,
> >
> > I noticed that JOSM have been commended in the last two Quality reports
> [1] [2] for the number and
> > indeed the quality of bugs you have found and logged against various
> early access releases.
> > Thanks again for your contribution to the quality of our releases!
> >
> > The big question is how do you do it ?
> >
> > We would love to hear more details about your test environment, what
> tests you run etc. Below is a
> > series of questions that come to mind, but there may be other details
> that you would like to share.
> >       • Builds
> >               • do you build your own OpenJDK builds, use Oracle's or
> other EA builds ?
> >       • Tests
> >               • What tests do you run
> >                       • do you run the JCK ?
> >                       • open regression tests ?
> >                       • other tests ?
> >                       • do you measure code coverage ?
> >               • Are the tests Automated or Manual ?
> >               • How frequently to do execute the tests
> >               • Are the results visible via an external link
> > Rgds, Rory
> >
> > [1] https://wiki.openjdk.java.net/display/quality/Quality+
> Outreach+report+September+2017
> > [2] https://wiki.openjdk.java.net/display/quality/Quality+
> Outreach+report+March+2017
> > --
> > Rgds,Rory O'Donnell
> > Quality Engineering Manager
> > Oracle EMEA , Dublin, Ireland
> >
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Client side testing

Vincent Privat
OK I tried to setup the JDK coverage, and this is NOT easy.
Right now I'm blocked by a bug in the Jenkins Jacoco plugin:
https://issues.jenkins-ci.org/browse/JENKINS-36168
Need to solve it first.
If I find how to setup JDK coverage for an Ant project, it will deserve
serious documentation. I will probably update JOSM wiki, but maybe it could
land somewhere on
https://wiki.openjdk.java.net/display/quality/Quality+Outreach for other
projects?

2017-11-06 23:51 GMT+01:00 Alexandre (Shura) Iline <
[hidden email]>:

>
> On Nov 6, 2017, at 2:48 PM, Vincent Privat <[hidden email]>
> wrote:
>
> Sure! Currently coverage is restricted to JOSM classes, for obvious
> performance reasons, as we run a full build after each commit.
> But I can add a parameter to change the monitored packages in our EA
> builds.
> Is it OK if I add java, javax, sun and com.sun ? Or do you prefer to
> restrict only to packages from java.desktop module?
>
>
> Thank you!
>
> Frankly, java.desktop is what I am more interested myself right now.
>
> java, javax, sun and com.sun would also cover what I am asking for, of
> course.
>
> Shura
>
>
>
>
> 2017-11-06 19:31 GMT+01:00 Alexandre (Shura) Iline <
> [hidden email]>:
>
>> This is great that you are touching the new builds, thank you, Vincent.
>>
>> I am not familiar with JOSM code, but the app is great! I have played
>> with it in the past.
>>
>> Do you think it would be possible, in addition to what you are doing, to
>> collect code coverage for the  JDK code? Specifically for the now existing
>> java.desktop module. I do not know much of JaCoCo - we are using JCov, as
>> you can imagine - so I do not know how easy it is. If it is easy, can you
>> flip the switch?
>>
>> Shura
>> > On Nov 3, 2017, at 8:33 AM, Vincent Privat <[hidden email]>
>> wrote:
>> >
>> > Hello Rory,
>> > Thank you :)
>> >
>> > Here is how we test various OpenJDK builds:
>> > 1.1: we use only EA builds from http://jdk.java.net/
>> > 2.1.1: no
>> > 2.1.2/2.1.3: we run all unit and integration tests of JOSM
>> > 2.1.4: yes, using jacoco
>> > 2.2: all tests are automated
>> > 2.3: it depends. During OpenJDK9 cycle we did not run a test series for
>> each weekly build, but rather once every two or three builds, unless we
>> were waiting for a specific correction from OpenJDK or another tool. At the
>> end of development, we were not waiting for any correction from OpenJDK but
>> were eagerly waiting for the updates of ASM and everything that as blocked
>> by the non-availability of ASM (in particular various Maven plugins)
>> > 2.4: Yes, they are visible here:
>> > - JOSM core: https://josm.openstreetmap.de/
>> jenkins/job/Java-EarlyAccess-JOSM/
>> > - JOSM community plugins: https://josm.openstreetmap.de/
>> jenkins/job/Java-EarlyAccess-JOSM-Plugins/
>> >
>> > I just have initiated JDK10 tests today, and we have 53 failures caused
>> by Byte Buddy not recognizing Java version 10:
>> >
>> > java.lang.IllegalArgumentException: Unknown Java version: 10
>> >       at nl.jqno.equalsverifier.internal.lib.bytebuddy.ClassFileVersi
>> on.ofJavaVersion(ClassFileVersion.java:135)
>> >
>> >
>> > I'm sure Rafael is aware of it. I will see how to update our tools
>> (equalsverifier here).
>> >
>> > Cheers,
>> > Vincent
>> >
>> > 2017-11-03 14:05 GMT+01:00 Rory O'Donnell <[hidden email]>:
>> >
>> > Hi Vincent,
>> >
>> > I noticed that JOSM have been commended in the last two Quality reports
>> [1] [2] for the number and
>> > indeed the quality of bugs you have found and logged against various
>> early access releases.
>> > Thanks again for your contribution to the quality of our releases!
>> >
>> > The big question is how do you do it ?
>> >
>> > We would love to hear more details about your test environment, what
>> tests you run etc. Below is a
>> > series of questions that come to mind, but there may be other details
>> that you would like to share.
>> >       • Builds
>> >               • do you build your own OpenJDK builds, use Oracle's or
>> other EA builds ?
>> >       • Tests
>> >               • What tests do you run
>> >                       • do you run the JCK ?
>> >                       • open regression tests ?
>> >                       • other tests ?
>> >                       • do you measure code coverage ?
>> >               • Are the tests Automated or Manual ?
>> >               • How frequently to do execute the tests
>> >               • Are the results visible via an external link
>> > Rgds, Rory
>> >
>> > [1] https://wiki.openjdk.java.net/display/quality/Quality+Outrea
>> ch+report+September+2017
>> > [2] https://wiki.openjdk.java.net/display/quality/Quality+Outrea
>> ch+report+March+2017
>> > --
>> > Rgds,Rory O'Donnell
>> > Quality Engineering Manager
>> > Oracle EMEA , Dublin, Ireland
>> >
>> >
>>
>>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Client side testing

Vincent Privat
If someone's paid at Oracle to work on Jenkins plugins, this bug is a good
candidate :)
(given the Hudson/Jenkins history I don't think so, but well, sometimes we
have good surprises, as I can see Jenkins is used to power
https://ci.adoptopenjdk.net/)

2017-11-07 22:18 GMT+01:00 Alexandre (Shura) Iline <
[hidden email]>:

> The issue is pretty generic and also pretty old. :(
>
> Thank you anyway.
>
> Let me know if there is anything we could help with on our side.
>
> Shuta
>
>
>
> On Nov 7, 2017, at 1:11 PM, Vincent Privat <[hidden email]>
> wrote:
>
> OK I tried to setup the JDK coverage, and this is NOT easy.
> Right now I'm blocked by a bug in the Jenkins Jacoco plugin:
> https://issues.jenkins-ci.org/browse/JENKINS-36168
> Need to solve it first.
> If I find how to setup JDK coverage for an Ant project, it will deserve
> serious documentation. I will probably update JOSM wiki, but maybe it could
> land somewhere on https://wiki.openjdk.java.net/display/quality/Quality+
> Outreach for other projects?
>
> 2017-11-06 23:51 GMT+01:00 Alexandre (Shura) Iline <
> [hidden email]>:
>
>>
>> On Nov 6, 2017, at 2:48 PM, Vincent Privat <[hidden email]>
>> wrote:
>>
>> Sure! Currently coverage is restricted to JOSM classes, for obvious
>> performance reasons, as we run a full build after each commit.
>> But I can add a parameter to change the monitored packages in our EA
>> builds.
>> Is it OK if I add java, javax, sun and com.sun ? Or do you prefer to
>> restrict only to packages from java.desktop module?
>>
>>
>> Thank you!
>>
>> Frankly, java.desktop is what I am more interested myself right now.
>>
>> java, javax, sun and com.sun would also cover what I am asking for, of
>> course.
>>
>> Shura
>>
>>
>>
>>
>> 2017-11-06 19:31 GMT+01:00 Alexandre (Shura) Iline <
>> [hidden email]>:
>>
>>> This is great that you are touching the new builds, thank you, Vincent.
>>>
>>> I am not familiar with JOSM code, but the app is great! I have played
>>> with it in the past.
>>>
>>> Do you think it would be possible, in addition to what you are doing, to
>>> collect code coverage for the  JDK code? Specifically for the now existing
>>> java.desktop module. I do not know much of JaCoCo - we are using JCov, as
>>> you can imagine - so I do not know how easy it is. If it is easy, can you
>>> flip the switch?
>>>
>>> Shura
>>> > On Nov 3, 2017, at 8:33 AM, Vincent Privat <[hidden email]>
>>> wrote:
>>> >
>>> > Hello Rory,
>>> > Thank you :)
>>> >
>>> > Here is how we test various OpenJDK builds:
>>> > 1.1: we use only EA builds from http://jdk.java.net/
>>> > 2.1.1: no
>>> > 2.1.2/2.1.3: we run all unit and integration tests of JOSM
>>> > 2.1.4: yes, using jacoco
>>> > 2.2: all tests are automated
>>> > 2.3: it depends. During OpenJDK9 cycle we did not run a test series
>>> for each weekly build, but rather once every two or three builds, unless we
>>> were waiting for a specific correction from OpenJDK or another tool. At the
>>> end of development, we were not waiting for any correction from OpenJDK but
>>> were eagerly waiting for the updates of ASM and everything that as blocked
>>> by the non-availability of ASM (in particular various Maven plugins)
>>> > 2.4: Yes, they are visible here:
>>> > - JOSM core: https://josm.openstreetmap.de/
>>> jenkins/job/Java-EarlyAccess-JOSM/
>>> > - JOSM community plugins: https://josm.openstreetmap.de/
>>> jenkins/job/Java-EarlyAccess-JOSM-Plugins/
>>> >
>>> > I just have initiated JDK10 tests today, and we have 53 failures
>>> caused by Byte Buddy not recognizing Java version 10:
>>> >
>>> > java.lang.IllegalArgumentException: Unknown Java version: 10
>>> >       at nl.jqno.equalsverifier.internal.lib.bytebuddy.ClassFileVersi
>>> on.ofJavaVersion(ClassFileVersion.java:135)
>>> >
>>> >
>>> > I'm sure Rafael is aware of it. I will see how to update our tools
>>> (equalsverifier here).
>>> >
>>> > Cheers,
>>> > Vincent
>>> >
>>> > 2017-11-03 14:05 GMT+01:00 Rory O'Donnell <[hidden email]>:
>>> >
>>> > Hi Vincent,
>>> >
>>> > I noticed that JOSM have been commended in the last two Quality
>>> reports [1] [2] for the number and
>>> > indeed the quality of bugs you have found and logged against various
>>> early access releases.
>>> > Thanks again for your contribution to the quality of our releases!
>>> >
>>> > The big question is how do you do it ?
>>> >
>>> > We would love to hear more details about your test environment, what
>>> tests you run etc. Below is a
>>> > series of questions that come to mind, but there may be other details
>>> that you would like to share.
>>> >       • Builds
>>> >               • do you build your own OpenJDK builds, use Oracle's or
>>> other EA builds ?
>>> >       • Tests
>>> >               • What tests do you run
>>> >                       • do you run the JCK ?
>>> >                       • open regression tests ?
>>> >                       • other tests ?
>>> >                       • do you measure code coverage ?
>>> >               • Are the tests Automated or Manual ?
>>> >               • How frequently to do execute the tests
>>> >               • Are the results visible via an external link
>>> > Rgds, Rory
>>> >
>>> > [1] https://wiki.openjdk.java.net/display/quality/Quality+Outrea
>>> ch+report+September+2017
>>> > [2] https://wiki.openjdk.java.net/display/quality/Quality+Outrea
>>> ch+report+March+2017
>>> > --
>>> > Rgds,Rory O'Donnell
>>> > Quality Engineering Manager
>>> > Oracle EMEA , Dublin, Ireland
>>> >
>>> >
>>>
>>>
>>
>>
>
>