Test First Rule

In the last section, we created a new rule package and wrote our first rule.

But the rule we wrote had bugs! In this section, we will look at how to use regle's built-in type checker and test utilities to help us find and fix them.

Type Checker

regle-cli comes with a built-in type checker to identify common type errors.

Let's see it in action. In a terminal, run:

$ regle compile tutorial

note

If you changed the rule package name, change tutorial to the directory containing main.tw from last section

You should see an output similar to the following:

$ regle compile tutorial

in file main on line 3

  | LET UserEmail: String = jsonExtract(EventData, "$.user.email");
3 | LET LoginOk = jsonExtract(EventData, "$.loginSuccess");
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Json feature `LoginOk` must be explicitly typed
  |
  | LET UserEmailToken = toToken("Email", UserEmail);

Ah ha. regle is telling us to provide type annotation for feature LoginOk. Let's update LoginOk definition to:

main.tw

LET LoginOk: Bool = jsonExtract(EventData, "$.loginSuccess");

Let's run the type checker again...

$ regle compile tutorial

$ regle compile tutorial

in file main on line 9

  | LET UserFailedLogInAttempt = getCount("failed_login", BY UserEmailToken, 5min);
  |
  | RULE "repeated_login_failure"
9 | LABEL UserEmail AS "potential_ato_target"
          ^^^^^^^^^ Expected Token, found String
  | WHERE UserFailedLogInAttempt < 2;
  |
  | # keep track of users who failed to login

Voila. A new error. Type checker found that we accidentally used UserEmail instead of UserEmailToken in the rules label clause.

Let's fix it. The rule definition in main.tw should now look like:

main.tw

...
RULE "repeated_login_failure" 
LABEL UserEmailToken AS "potential_ato_target"
WHERE UserFailedLogInAttempt < 2;
...

Let's run the type checker one more time...and no errors!

But we still haven't fixed the our logic error in our rule defintion. Let's add a test for that.

Test in regle

Let's take a look at the generated __test__ directory we skipped last time.

It contains two files, test.tw and test_event.json.

test_event.json

{
    "name": "test_event",
    "data": { }
}

test_event.json is a placeholder JSON event similar to one that regle would process in production.

test.tw

_loadEvent("test_event.json");

LET Result = _execute();
LET Labels: Json = jsonExtract(Result, "$.labels");

ASSERT len(Labels) == 0;

LET Result2 = _execute();
LET Labels2: Json = jsonExtract(Result2, "$.labels");

ASSERT len(Labels) == 0;

In test.tw, we load a JSON event via _loadEvent function by supplying it with an event path relative to __test__ directory.

After the test event is loaded, we call _execute function to run the tutorial package code we wrote.

_execute returns the evaluation result in JSON (see complete schema). All we need to know right now is that it contains a list of labels generated at $.labels, which we can use to test our rule repeated_login_failure.

We can call _execute function multiple times to simulate processing a continuous stream of events. We can also update the JSON event by calling _loadEvent again.

Test Our Package

Let's update test_evet.json to:

test_event.json

{
  "name": "login",
  "data": {
    "user": {
      "email": "[email protected]"
    },
    "ip": "127.0.0.1",
    "loginSuccess": false
  }
}

Now update the test.tw to (with the change highlighted):

test.tw

_loadEvent("test_event.json");

LET Result = _execute();
LET Labels : Json = jsonExtract(Result, "$.labels");
# check we extracted the right field off the event JSON
ASSERT pkg.UserEmail == "[email protected]";
# check that UserFailedLogInAttempt is zero 
ASSERT pkg.UserFailedLogInAttempt == 0;
ASSERT len(Labels) == 0;

LET Result2 = _execute();
LET Labels2 : Json = jsonExtract(Result2, "$.labels");
ASSERT pkg.UserFailedLogInAttempt == 1;
ASSERT len(Labels2) == 1;

Inside the test files (any tw files under __test__ directory), we can refer to features defined in our package via pkg.<FEATURE_NAME> syntax.

We added an ASSERT on line 6 to check the email is as expected. On line 7, we verify the counter returned 0 since this is the first time we observed a login failure. Similarly, we assert the number of labels generated is zero on line 9.

When we evaluate the same event again by calling _execute a second time, we are simulating the same user failing to login in short succession. During this run, we expect pkg.UserFailedLogInAttempt to be 1 as this is the second time we saw the same email failing login (counter is read before the current event processing increased the counter). We also expect to generate 1 label, "potential_ato_target". That's the ASSERT statement on line 9.

Let's run the test:

$ regle test tutorial

in file /tmp/pkg1/__test__/test on line 8

  |
  | ASSERT pkg.UserEmail == "[email protected]";
  | ASSERT pkg.UserFailedLogInAttempt == 0;
8 | ASSERT len(Labels) == 0;
    ^^^^^^^^^^^^^^^^^^^^^^^^ Assert failure
  |
  | LET Result2 = _execute();
  | LET Labels2 : Json = jsonExtract(Result2, "$.labels");

Our test failed since we accidentally reversed our comparison statement (see here to refresh your memory).

Let's fix our main.tw by updating line 10 to:

main.tw

...
WHERE UserFailedLogInAttempt >= 1;
...

Let's run the test again:

$ regle test tutorial

...and no errors!