Hi all,
this is a project I did for the testing course this semester. I hope
it could shed a light for testing aspect of dhis2.
Notice: the case was built based on the assumption that the system
specification requires name of orgunit, group, and group set to be
have between 2-255 characters. Though dhis2 implemented it with text
field max char = 160.
Thanh
project report final.pdf (372 KB)
Hi Thanh
We did receive it and i had a quick read through. Its great to see
work being done on testing. we need lots of it.
Two comments/questions (to prove that I read it!):
1. I am not entirely convinced by your replacement of TSL with Excel.
TSL is a formal grammar. It has limitations which have been
described elsewhere but I don't think that the typing of '[' has ever
really featured highly 
I think you may be confusing the
specification with the authoring tool when you say that your revised
TSL is to "use excel". So on the theory side I'd be a bit
uncomfortable with this aspect of your paper.
(BTW did you look at schematron as an alternative grammar for
expressing tests? I know one of the criticisms of TSL is that it does
not allow for a very literate human-readable approach. schematron
allows for a human readable assertions and - though xpath is usually
used for schema validation - tests can be implemented in any language.
Perhaps an off-the wall thought ...)
2. I am not very clear - perhaps didn't read carefully enough - how
you generate the actual tests from the source document. It looks to
me that it requires quite a significant understanding of the
application , the classes etc. I guess this is where understanding
the grammar, if any, is important. Could you generate selenium tests
from it? This strikes me as the most black-box kind of testing.
How easy would it be to use your tool to do wider test coverage of
dhis? Would be a good thing.
Cheers
Bob
On 2 June 2010 12:03, Ngoc Thanh Nguyen <thanh.hispvietnam@gmail.com> wrote:
Hi all,
this is a project I did for the testing course this semester. I hope
it could shed a light for testing aspect of dhis2.Notice: the case was built based on the assumption that the system
specification requires name of orgunit, group, and group set to be
have between 2-255 characters. Though dhis2 implemented it with text
field max char = 160.Thanh
_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
Hi Bob,
Thanks for the comments. I have some replies below.
Hi Thanh
We did receive it and i had a quick read through. Its great to see
work being done on testing. we need lots of it.Two comments/questions (to prove that I read it!):
1. I am not entirely convinced by your replacement of TSL with Excel.
TSL is a formal grammar. It has limitations which have been
described elsewhere but I don't think that the typing of '[' has ever
really featured highly
specification with the authoring tool when you say that your revised
TSL is to "use excel". So on the theory side I'd be a bit
uncomfortable with this aspect of your paper.
(BTW did you look at schematron as an alternative grammar for
expressing tests? I know one of the criticisms of TSL is that it does
not allow for a very literate human-readable approach. schematron
allows for a human readable assertions and - though xpath is usually
used for schema validation - tests can be implemented in any language.
Perhaps an off-the wall thought ...)
TSL in my paper is about the specification developed by Ostrand 1988.
They developed this specification within the method they called
Category Partition - a combination between Equivalence Partition and
Boundary Analysis. This specification as I feel is a bit complicated
because it required typing [ or ] character. My revision aims to save
time for typing this and also make it readable for a computer program.
I selected Excel because of its popularity and easy-to-use.
CPM is the most popular techniques in black box testing. Cause graph
effect is another method but it requires to draw graphical diagram so
it seems impossible in practice.
XPath or Schematron seems to be out of the scope of CPM.
2. I am not very clear - perhaps didn't read carefully enough - how
you generate the actual tests from the source document. It looks to
me that it requires quite a significant understanding of the
application , the classes etc. I guess this is where understanding
the grammar, if any, is important. Could you generate selenium tests
from it? This strikes me as the most black-box kind of testing.
No, CPM can not generate actual test cases and test data. It can
generate test frame (a combination of different input values). Tester
has to provide the test data, hence, to build test case. I find it
hard to have any automation tool to generate test cases (with test
data) automatically. And my tool just does a simple job: 1) read the
Excel file containing category and partition, 2) combination based on
their properties and constraints, 3) and build the test frames based
on the combination.
How easy would it be to use your tool to do wider test coverage of
dhis? Would be a good thing.
Here we come with the test coverage. Black box is different from white
box. It can not produce any code coverage (statement, block, branch,
predicate etc). Not at all. It can not be used with mutation. It seems
there is no way to assess whether a test set is adequate - good
enough. Any one have better idea can correct me on this.
But I am very confident that if we follow the CPM strictly, together
with the support from my tool, i.e. revise the category and partition
building back and forth, we can end up with a 100% test coverage.
The challenge is, for web layer of DHIS2, we could not apply white box
testing. This is partial reason why there is no unit testing for
action class. It is so difficult to mock all the session and
interceptor at least in DHIS2. We have only one choice, that is black
box testing. And so far, we have not done any systematic testing on
web modules. Our current approach is more ad-hoc.
More input needed.
Thanh
On Thu, Jun 3, 2010 at 3:40 PM, Bob Jolliffe <bobjolliffe@gmail.com> wrote:
Cheers
BobOn 2 June 2010 12:03, Ngoc Thanh Nguyen <thanh.hispvietnam@gmail.com> wrote:
Hi all,
this is a project I did for the testing course this semester. I hope
it could shed a light for testing aspect of dhis2.Notice: the case was built based on the assumption that the system
specification requires name of orgunit, group, and group set to be
have between 2-255 characters. Though dhis2 implemented it with text
field max char = 160.Thanh
_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
Hi
Hi Bob,
Thanks for the comments. I have some replies below.
Hi Thanh
We did receive it and i had a quick read through. Its great to see
work being done on testing. we need lots of it.Two comments/questions (to prove that I read it!):
1. I am not entirely convinced by your replacement of TSL with Excel.
TSL is a formal grammar. It has limitations which have been
described elsewhere but I don't think that the typing of '[' has ever
really featured highly
specification with the authoring tool when you say that your revised
TSL is to "use excel". So on the theory side I'd be a bit
uncomfortable with this aspect of your paper.
(BTW did you look at schematron as an alternative grammar for
expressing tests? I know one of the criticisms of TSL is that it does
not allow for a very literate human-readable approach. schematron
allows for a human readable assertions and - though xpath is usually
used for schema validation - tests can be implemented in any language.
Perhaps an off-the wall thought ...)TSL in my paper is about the specification developed by Ostrand 1988.
They developed this specification within the method they called
Category Partition - a combination between Equivalence Partition and
Boundary Analysis. This specification as I feel is a bit complicated
because it required typing [ or ] character. My revision aims to save
time for typing this and also make it readable for a computer program.
I selected Excel because of its popularity and easy-to-use.
Yes I understand all that. So you have selected excel as a tool for
authoring your test cases. Thats fine and I'm sure its a good tool
for the job for the reasons you describe. The point is that TSL is a
language for expressing them. What do you express? An "excel file".
I really don't think thats the the same thing at all. I guess what is
missing is some way to describe those spreadsheets of yours so that
they can be considered as a sort of language like TSL.
CPM is the most popular techniques in black box testing. Cause graph
effect is another method but it requires to draw graphical diagram so
it seems impossible in practice.XPath or Schematron seems to be out of the scope of CPM.
Not at all. See
CSDL | IEEE Computer Society for
example.
I agree XPath is well out of scope, but schematron is not bound to use
XPath. Just so happens that 99% of the time it does because it is
primarily used for testing xml validity. But I guess there is nothing
in principle preventing the actual tests to be implemented as say
javascript, or even selenium scripts.
2. I am not very clear - perhaps didn't read carefully enough - how
you generate the actual tests from the source document. It looks to
me that it requires quite a significant understanding of the
application , the classes etc. I guess this is where understanding
the grammar, if any, is important. Could you generate selenium tests
from it? This strikes me as the most black-box kind of testing.No, CPM can not generate actual test cases and test data. It can
generate test frame (a combination of different input values). Tester
has to provide the test data, hence, to build test case. I find it
hard to have any automation tool to generate test cases (with test
data) automatically. And my tool just does a simple job: 1) read the
Excel file containing category and partition, 2) combination based on
their properties and constraints, 3) and build the test frames based
on the combination.
OK. Now I understand.
How easy would it be to use your tool to do wider test coverage of
dhis? Would be a good thing.Here we come with the test coverage. Black box is different from white
box. It can not produce any code coverage (statement, block, branch,
predicate etc). Not at all. It can not be used with mutation. It seems
there is no way to assess whether a test set is adequate - good
enough. Any one have better idea can correct me on this.But I am very confident that if we follow the CPM strictly, together
with the support from my tool, i.e. revise the category and partition
building back and forth, we can end up with a 100% test coverage.
Sounds good.
The challenge is, for web layer of DHIS2, we could not apply white box
testing. This is partial reason why there is no unit testing for
action class. It is so difficult to mock all the session and
interceptor at least in DHIS2. We have only one choice, that is black
box testing. And so far, we have not done any systematic testing on
web modules. Our current approach is more ad-hoc.
This is where I was suggesting something like selenium might be
useful. Does anybody know of any formal testing language for web
applications? I am sure there is bound to be something.
Thanh, thanks for sharing this. Now I must get back to work ... bloody spring.
Regards
Bob
On 3 June 2010 10:50, Ngoc Thanh Nguyen <thanh.hispvietnam@gmail.com> wrote:
On Thu, Jun 3, 2010 at 3:40 PM, Bob Jolliffe <bobjolliffe@gmail.com> wrote:
More input needed.
ThanhCheers
BobOn 2 June 2010 12:03, Ngoc Thanh Nguyen <thanh.hispvietnam@gmail.com> wrote:
Hi all,
this is a project I did for the testing course this semester. I hope
it could shed a light for testing aspect of dhis2.Notice: the case was built based on the assumption that the system
specification requires name of orgunit, group, and group set to be
have between 2-255 characters. Though dhis2 implemented it with text
field max char = 160.Thanh
_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
Hi Bob,
Thanks for the feedbacks written in your busy time :-). But I have a
small comments on Selenium. Please read below.
Hi
Hi Bob,
Thanks for the comments. I have some replies below.
Hi Thanh
We did receive it and i had a quick read through. Its great to see
work being done on testing. we need lots of it.Two comments/questions (to prove that I read it!):
1. I am not entirely convinced by your replacement of TSL with Excel.
TSL is a formal grammar. It has limitations which have been
described elsewhere but I don't think that the typing of '[' has ever
really featured highly
specification with the authoring tool when you say that your revised
TSL is to "use excel". So on the theory side I'd be a bit
uncomfortable with this aspect of your paper.
(BTW did you look at schematron as an alternative grammar for
expressing tests? I know one of the criticisms of TSL is that it does
not allow for a very literate human-readable approach. schematron
allows for a human readable assertions and - though xpath is usually
used for schema validation - tests can be implemented in any language.
Perhaps an off-the wall thought ...)TSL in my paper is about the specification developed by Ostrand 1988.
They developed this specification within the method they called
Category Partition - a combination between Equivalence Partition and
Boundary Analysis. This specification as I feel is a bit complicated
because it required typing [ or ] character. My revision aims to save
time for typing this and also make it readable for a computer program.
I selected Excel because of its popularity and easy-to-use.Yes I understand all that. So you have selected excel as a tool for
authoring your test cases. Thats fine and I'm sure its a good tool
for the job for the reasons you describe. The point is that TSL is a
language for expressing them. What do you express? An "excel file".
I really don't think thats the the same thing at all. I guess what is
missing is some way to describe those spreadsheets of yours so that
they can be considered as a sort of language like TSL.CPM is the most popular techniques in black box testing. Cause graph
effect is another method but it requires to draw graphical diagram so
it seems impossible in practice.XPath or Schematron seems to be out of the scope of CPM.
Not at all. See
CSDL | IEEE Computer Society for
example.I agree XPath is well out of scope, but schematron is not bound to use
XPath. Just so happens that 99% of the time it does because it is
primarily used for testing xml validity. But I guess there is nothing
in principle preventing the actual tests to be implemented as say
javascript, or even selenium scripts.2. I am not very clear - perhaps didn't read carefully enough - how
you generate the actual tests from the source document. It looks to
me that it requires quite a significant understanding of the
application , the classes etc. I guess this is where understanding
the grammar, if any, is important. Could you generate selenium tests
from it? This strikes me as the most black-box kind of testing.No, CPM can not generate actual test cases and test data. It can
generate test frame (a combination of different input values). Tester
has to provide the test data, hence, to build test case. I find it
hard to have any automation tool to generate test cases (with test
data) automatically. And my tool just does a simple job: 1) read the
Excel file containing category and partition, 2) combination based on
their properties and constraints, 3) and build the test frames based
on the combination.OK. Now I understand.
How easy would it be to use your tool to do wider test coverage of
dhis? Would be a good thing.Here we come with the test coverage. Black box is different from white
box. It can not produce any code coverage (statement, block, branch,
predicate etc). Not at all. It can not be used with mutation. It seems
there is no way to assess whether a test set is adequate - good
enough. Any one have better idea can correct me on this.But I am very confident that if we follow the CPM strictly, together
with the support from my tool, i.e. revise the category and partition
building back and forth, we can end up with a 100% test coverage.Sounds good.
The challenge is, for web layer of DHIS2, we could not apply white box
testing. This is partial reason why there is no unit testing for
action class. It is so difficult to mock all the session and
interceptor at least in DHIS2. We have only one choice, that is black
box testing. And so far, we have not done any systematic testing on
web modules. Our current approach is more ad-hoc.This is where I was suggesting something like selenium might be
useful. Does anybody know of any formal testing language for web
applications? I am sure there is bound to be something.
I have tried Selenium, indeed, I have built several test cases on
Selenium and it run quite well. I also tried sahi, another web
automation test tool. These tool help tester to record their tests and
save in different format, hence they can be run later without manual
re-running. HOWEVER, what to be recoded in Selenium is really a
question. And CPM comes to fulfill this space.
So I recommend the following process: building test specification
(category, partition, properties, constrainst) --> use the tool to
generate the test frame (or test case if you want to call) --> record
these test cases using Selenium --> enjoy the automation forever.
Thanh
On Thu, Jun 3, 2010 at 5:17 PM, Bob Jolliffe <bobjolliffe@gmail.com> wrote:
On 3 June 2010 10:50, Ngoc Thanh Nguyen <thanh.hispvietnam@gmail.com> wrote:
On Thu, Jun 3, 2010 at 3:40 PM, Bob Jolliffe <bobjolliffe@gmail.com> wrote:
Thanh, thanks for sharing this. Now I must get back to work ... bloody spring.
Regards
BobMore input needed.
ThanhCheers
BobOn 2 June 2010 12:03, Ngoc Thanh Nguyen <thanh.hispvietnam@gmail.com> wrote:
Hi all,
this is a project I did for the testing course this semester. I hope
it could shed a light for testing aspect of dhis2.Notice: the case was built based on the assumption that the system
specification requires name of orgunit, group, and group set to be
have between 2-255 characters. Though dhis2 implemented it with text
field max char = 160.Thanh
_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp