[PATCH v3 0/1] lib: Convert UUID runtime test to KUnit

André Almeida posted 1 patch 5 days, 15 hours ago
Failed in applying to current master (apply log)
lib/Kconfig.debug |  11 +++-
lib/Makefile      |   2 +-
lib/test_uuid.c   | 137 +++++++++++++++++++---------------------------
3 files changed, 67 insertions(+), 83 deletions(-)

[PATCH v3 0/1] lib: Convert UUID runtime test to KUnit

Posted by André Almeida 5 days, 15 hours ago
Hi,

This patch converts existing UUID runtime test to use KUnit framework.

Below, there's a comparison between the old output format and the new
one. Keep in mind that even if KUnit seems very verbose, this is the
corner case where _every_ test has failed.

* This is how the current output looks like in success:

  test_uuid: all 18 tests passed

* And when it fails:

  test_uuid: conversion test #1 failed on LE data: 'c33f4995-3701-450e-9fbf-206a2e98e576'
  test_uuid: cmp test #2 failed on LE data: 'c33f4995-3701-450e-9fbf-206a2e98e576'
  test_uuid: cmp test #2 actual data: 'c33f4995-3701-450e-9fbf-206a2e98e576'
  test_uuid: conversion test #3 failed on BE data: 'c33f4995-3701-450e-9fbf-206a2e98e576'
  test_uuid: cmp test #4 failed on BE data: 'c33f4995-3701-450e-9fbf-206a2e98e576'
  test_uuid: cmp test #4 actual data: 'c33f4995-3701-450e-9fbf-206a2e98e576'
  test_uuid: conversion test #5 failed on LE data: '64b4371c-77c1-48f9-8221-29f054fc023b'
  test_uuid: cmp test #6 failed on LE data: '64b4371c-77c1-48f9-8221-29f054fc023b'
  test_uuid: cmp test #6 actual data: '64b4371c-77c1-48f9-8221-29f054fc023b'
  test_uuid: conversion test #7 failed on BE data: '64b4371c-77c1-48f9-8221-29f054fc023b'
  test_uuid: cmp test #8 failed on BE data: '64b4371c-77c1-48f9-8221-29f054fc023b'
  test_uuid: cmp test #8 actual data: '64b4371c-77c1-48f9-8221-29f054fc023b'
  test_uuid: conversion test #9 failed on LE data: '0cb4ddff-a545-4401-9d06-688af53e7f84'
  test_uuid: cmp test #10 failed on LE data: '0cb4ddff-a545-4401-9d06-688af53e7f84'
  test_uuid: cmp test #10 actual data: '0cb4ddff-a545-4401-9d06-688af53e7f84'
  test_uuid: conversion test #11 failed on BE data: '0cb4ddff-a545-4401-9d06-688af53e7f84'
  test_uuid: cmp test #12 failed on BE data: '0cb4ddff-a545-4401-9d06-688af53e7f84'
  test_uuid: cmp test #12 actual data: '0cb4ddff-a545-4401-9d06-688af53e7f84'
  test_uuid: negative test #13 passed on wrong LE data: 'c33f4995-3701-450e-9fbf206a2e98e576 '
  test_uuid: negative test #14 passed on wrong BE data: 'c33f4995-3701-450e-9fbf206a2e98e576 '
  test_uuid: negative test #15 passed on wrong LE data: '64b4371c-77c1-48f9-8221-29f054XX023b'
  test_uuid: negative test #16 passed on wrong BE data: '64b4371c-77c1-48f9-8221-29f054XX023b'
  test_uuid: negative test #17 passed on wrong LE data: '0cb4ddff-a545-4401-9d06-688af53e'
  test_uuid: negative test #18 passed on wrong BE data: '0cb4ddff-a545-4401-9d06-688af53e'
  test_uuid: failed 18 out of 18 tests


* Now, here's how it looks like with KUnit:

  ======== [PASSED] uuid ========
  [PASSED] uuid_correct_be
  [PASSED] uuid_correct_le
  [PASSED] uuid_wrong_be
  [PASSED] uuid_wrong_le

* And if every test fail with KUnit:

  ======== [FAILED] uuid ========
  [FAILED] uuid_correct_be
      # uuid_correct_be: ASSERTION FAILED at lib/test_uuid.c:57
      Expected uuid_parse(data->uuid, &be) == 1, but
          uuid_parse(data->uuid, &be) == 0
  
  failed to parse 'c33f4995-3701-450e-9fbf-206a2e98e576'
      # uuid_correct_be: not ok 1 - c33f4995-3701-450e-9fbf-206a2e98e576
      # uuid_correct_be: ASSERTION FAILED at lib/test_uuid.c:57
      Expected uuid_parse(data->uuid, &be) == 1, but
          uuid_parse(data->uuid, &be) == 0
  
  failed to parse '64b4371c-77c1-48f9-8221-29f054fc023b'
      # uuid_correct_be: not ok 2 - 64b4371c-77c1-48f9-8221-29f054fc023b
      # uuid_correct_be: ASSERTION FAILED at lib/test_uuid.c:57
      Expected uuid_parse(data->uuid, &be) == 1, but
          uuid_parse(data->uuid, &be) == 0
  
  failed to parse '0cb4ddff-a545-4401-9d06-688af53e7f84'
      # uuid_correct_be: not ok 3 - 0cb4ddff-a545-4401-9d06-688af53e7f84
      not ok 1 - uuid_correct_be
  
  [FAILED] uuid_correct_le
      # uuid_correct_le: ASSERTION FAILED at lib/test_uuid.c:46
      Expected guid_parse(data->uuid, &le) == 1, but
          guid_parse(data->uuid, &le) == 0
  
  failed to parse 'c33f4995-3701-450e-9fbf-206a2e98e576'
      # uuid_correct_le: not ok 1 - c33f4995-3701-450e-9fbf-206a2e98e576
      # uuid_correct_le: ASSERTION FAILED at lib/test_uuid.c:46
      Expected guid_parse(data->uuid, &le) == 1, but
          guid_parse(data->uuid, &le) == 0
  
  failed to parse '64b4371c-77c1-48f9-8221-29f054fc023b'
      # uuid_correct_le: not ok 2 - 64b4371c-77c1-48f9-8221-29f054fc023b
      # uuid_correct_le: ASSERTION FAILED at lib/test_uuid.c:46
      Expected guid_parse(data->uuid, &le) == 1, but
          guid_parse(data->uuid, &le) == 0
  
  failed to parse '0cb4ddff-a545-4401-9d06-688af53e7f84'
      # uuid_correct_le: not ok 3 - 0cb4ddff-a545-4401-9d06-688af53e7f84
      not ok 2 - uuid_correct_le
  
  [FAILED] uuid_wrong_be
      # uuid_wrong_be: ASSERTION FAILED at lib/test_uuid.c:77
      Expected uuid_parse(*data, &be) == 0, but
          uuid_parse(*data, &be) == -22
  
  parsing of 'c33f4995-3701-450e-9fbf206a2e98e576 ' should've failed
      # uuid_wrong_be: not ok 1 - c33f4995-3701-450e-9fbf206a2e98e576
      # uuid_wrong_be: ASSERTION FAILED at lib/test_uuid.c:77
      Expected uuid_parse(*data, &be) == 0, but
          uuid_parse(*data, &be) == -22
  
  parsing of '64b4371c-77c1-48f9-8221-29f054XX023b' should've failed
      # uuid_wrong_be: not ok 2 - 64b4371c-77c1-48f9-8221-29f054XX023b
      # uuid_wrong_be: ASSERTION FAILED at lib/test_uuid.c:77
      Expected uuid_parse(*data, &be) == 0, but
          uuid_parse(*data, &be) == -22
  
  parsing of '0cb4ddff-a545-4401-9d06-688af53e' should've failed
      # uuid_wrong_be: not ok 3 - 0cb4ddff-a545-4401-9d06-688af53e
      not ok 3 - uuid_wrong_be
  
  [FAILED] uuid_wrong_le
      # uuid_wrong_le: ASSERTION FAILED at lib/test_uuid.c:68
      Expected guid_parse(*data, &le) == 0, but
          guid_parse(*data, &le) == -22
  
  parsing of 'c33f4995-3701-450e-9fbf206a2e98e576 ' should've failed
      # uuid_wrong_le: not ok 1 - c33f4995-3701-450e-9fbf206a2e98e576
      # uuid_wrong_le: ASSERTION FAILED at lib/test_uuid.c:68
      Expected guid_parse(*data, &le) == 0, but
          guid_parse(*data, &le) == -22
  
  parsing of '64b4371c-77c1-48f9-8221-29f054XX023b' should've failed
      # uuid_wrong_le: not ok 2 - 64b4371c-77c1-48f9-8221-29f054XX023b
      # uuid_wrong_le: ASSERTION FAILED at lib/test_uuid.c:68
      Expected guid_parse(*data, &le) == 0, but
          guid_parse(*data, &le) == -22
  
  parsing of '0cb4ddff-a545-4401-9d06-688af53e' should've failed
      # uuid_wrong_le: not ok 3 - 0cb4ddff-a545-4401-9d06-688af53e
      not ok 4 - uuid_wrong_le

Changes from v2:
 - Clarify in commit message the new test cases setup
v2: https://lore.kernel.org/lkml/20210609233730.164082-1-andrealmeid@collabora.com/

Changes from v1:
 - Test suite name: uuid_test -> uuid
 - Config name: TEST_UUID -> UUID_KUNIT_TEST
 - Config entry in the Kconfig file left where it is
 - Converted tests to use _MSG variant
v1: https://lore.kernel.org/lkml/20210605215215.171165-1-andrealmeid@collabora.com/

André Almeida (1):
  lib: Convert UUID runtime test to KUnit

 lib/Kconfig.debug |  11 +++-
 lib/Makefile      |   2 +-
 lib/test_uuid.c   | 137 +++++++++++++++++++---------------------------
 3 files changed, 67 insertions(+), 83 deletions(-)

-- 
2.31.1

Re: [PATCH v3 0/1] lib: Convert UUID runtime test to KUnit

Posted by Andy Shevchenko 4 days, 22 hours ago
On Thu, Jun 10, 2021 at 01:39:58PM -0300, André Almeida wrote:
> Hi,
> 
> This patch converts existing UUID runtime test to use KUnit framework.
> 
> Below, there's a comparison between the old output format and the new
> one. Keep in mind that even if KUnit seems very verbose, this is the
> corner case where _every_ test has failed.

Btw, do we have test coverage statistics?

I mean since we reduced 18 test cases to 12, do we still have the same / better
test coverage?

-- 
With Best Regards,
Andy Shevchenko


Re: [PATCH v3 0/1] lib: Convert UUID runtime test to KUnit

Posted by André Almeida 4 days, 21 hours ago
Hi Andy,

Às 06:55 de 11/06/21, Andy Shevchenko escreveu:
> On Thu, Jun 10, 2021 at 01:39:58PM -0300, André Almeida wrote:
>> Hi,
>>
>> This patch converts existing UUID runtime test to use KUnit framework.
>>
>> Below, there's a comparison between the old output format and the new
>> one. Keep in mind that even if KUnit seems very verbose, this is the
>> corner case where _every_ test has failed.
> 
> Btw, do we have test coverage statistics?
> 
> I mean since we reduced 18 test cases to 12, do we still have the same / better
> test coverage?
> 
I don't think we have automated statistics, but I can assure you that
the coverage it's exactly the same. We are testing two correlated
functions with the same input, in a single test case, instead of having
a single case for each one, so that's why the number of cases is reduced.

For example, instead of:

	total_tests++;
	if (guid_parse(data->uuid, &le))


	total_tests++;
	if (!guid_equal(&data->le, &le))

We now have:

	KUNIT_ASSERT_EQ(guid_parse(data->guid, &le), 0)
	KUNIT_EXPECT_TRUE(guid_equal(&data->le, &le))

That will count as a single test.