When I started using Codewars, there were only a handful of Haskell warriors, and Haskell support was only recent. While I was eager to solve the Haskell katas, there weren't many of them, so I started translating. I've translated about 23% of my solved katas, either to Haskell or to another language.
And while Codewars provides some documentation, I often ended up answering the same questions in the comments
To prevent repeating myself over and over again, and to have some kind of reference for some nifty tricks, I've started writing this "guide". And while its GitHub repository is called "codewars-rules", it's meant as a light-hearted lecture for the interested reader.
Therefore, it's dismissed as "The conjured Codewars codex". What can I say, I like arbitrary alliterations. The repository's description is a reference on QuickCheck's Arbitrary.
You've read the previous section and therefore know that "rules" are more or less meant as "tips and tricks". But to make this a little bit more distinct, let's quote a (fictitious) pirate:
[T]he code is more what you'd call "guidelines" than actual rules.
-- Barbossa in "Pirates of the Caribbean: The Curse of the Black Pearl"
This also means that whenever you read the word "rules" in this document, you can simply substitute it by "tips" or "guidelines". Heave ho!
If you find an error, have a better idea, or think that all of this is a pile of unicorn misery, feel free to create an issue or a pull-request. Have a look at the LICENCE.
Hi all. I'm a random Codewars user. I'm not affiliated to Entrefuse Inc. (the company behind Codewars) in any way, except for being a user of their product. So if you encounter any Codewars specific problem, make sure to head over to their issue tracker as I cannot change anything. Thanks!
© 2015 Benjamin Kästner. This document is licenced under the Creative Commons Attribution-ShareAlike 4.0 International Public License. For a list of all contributors, check the contributors on GitHub.
You want to write a kata? That's great! You've probably already solved a bunch of them, found issues and participated in the beta process. If you haven't, I strongly suggest you to check katas in beta before you start your own. It gives you great insight about what you might forget in your tests or in your description. And since issues get resolved, but not deleted, you can check what kind of issues your favourite kata had during its early days or beta.
So please, check things out. We'll wait here.
You're back? Great! Now you're ready to write your own kata. Well, almost. There's another page, hidden in the inner depths of the forums: the kata best practices. You should also read them, as they provide additional insight.
Now that you've read all that's available on Codewars itself, here are some tips and remarks you might find handy or want to keep in mind.
When your kata is out of beta, the typical user will solve it in their preferred language. They will either be fascinated and solve your kata multiple times (which is rather unlikely, sorry), or go on. Therefore, your kata should shine in that language.
However, this concludes that you may need to change return types in some circumstances or the input. For example, JavaScript, Python, Ruby and CoffeeScript allow a variable number of arguments (JS, CS: arguments), while C (without varlist), C++ (without variadic templates), Haskell, Java (without generics) and some other don't. Furthermore, some languages (like Haskell) don't allow list of heterogeneous types, which rules out some katas entirely.
And this is fine. If you need to use serious magic just to create a translation to another language, consider whether the user will enjoy the kata, or whether he will disdain the hideous abomination. Always remember to use your language's idiomatic code. And if someone proposes a translation to your kata and you think it is not really idiomatic, ask them.
See also the section on translations.
However, in some cases, it is better to strife away from idiomatic code, especially if it makes the kata a lot easier in the other language. It devalues the achievements of those who solved the harder version. In those cases make sure to block the library functions (see below).
But consistency should be taken in measurements. If this leads to completely unidiomatic code during your translation, consider creating a completely new kata instead.
Every language has its quirks. Some of them get discussed in the later sections. However, there's one thing that should get mentioned at this point: make sure to use your language's proposed naming convention, if there exists one.
This includes simple character cases like capitalized class names (almost all object oriented languages) or method names (CSharp), and whether a multi word function should be written with underscores (also called snake_case, e.g. my_awesome_function, Ruby, Python) or in camelCase (Haskell, JavaScript CoffeeScript).
This obviously concludes that the function will differ in different languages. But then again, remember the previous rule, a user solves a kata in a single language, not many. Make the user feel at home.
By the way, if you use abbreviations for your function's name, make sure that they are common. Otherwise just use the full name.
While elaborate formulations may fancy ferocious fans of funny alliterations (or similar), they're usually hard to read. If you write a kata that is meant for beginners, use simple words. Keep sentences short. Remove all run-on sentences come on you want to avoid that kind of sentence in your kata description those sentences are bland and hard to comprehend.
Essentially use the same kind of language you would like to read in a published programming book. It is hard to get right, but it is worth the additional time.
If you provide a story, break the description into a story and a task. That way, a reader can enjoy the story without all those xs , is and numbers. Save that for your task.
Even with proper grammar and spelling your kata can be hard to read.
Especially if you use arbitrary newlines or code without proper formatting, like
# do_something(1) should return "Hello world" do_something(1) == "Hello world"
or similar. That's not really something your readers enjoy. Instead, use proper markdown:
``` python
# do_something(1) should return "Hello world"
do_something(1) = "Hello world"
```# do_something(1) should return "Hello world"
do_something(1) = "Hello world"This looks a lot nicer than the previous one.
While it might seem tempting to use plain HTML and CSS, it's not future proof, as the Codewars style might change. Learning GitHub flavoured markdown doesn't take long so make sure to read it.
Also make sure that your text isn't ragged too much. Getting typographic features right is hard and can take a while. Just play around with different paragraph lengths, headings and tables. But always format your code. You don't want to have something like the aforementioned do_something in your description.
Types are everywhere. Even if you use dynamic typed languages, you've at least encountered some quirks of your language, like "12345" + 1 in JavaScript, or "xy isn't a ab" in Python or Ruby.
There are basically two things that can go wrong with types: you can ask the user for the wrong return type, or you accidentally use the wrong type in your reference solution.
The second one is a lot harder to notice, so lets keep this for later. Instead let us look at one often used wrong return type first: the string.
Have a look at the following function:
static public string do_the_thing(string){
// ...
}What does this function do? It takes a string, and it returns a string. But it's not clear what happens there. Does the string need to follow a certain pattern, like /\s*(\d+\s+)+\s*/g? Is the string actually an array or a list, but as a text?
Compare the function above with the function below:
static public int sum_all_things(int[] numbers){
// ...
}The function's type already tells you that you'll get a bunch of numbers, and if this compiles, the caller probably gave you a bunch of numbers (or no number or null, but that's not important).
Using a string has several drawbacks, some of them mentioned above:
true and false. It doesn't give you any more information than the boolean."a value" instead of "a value", he might not notice this.That being said, if you provide a fitting story or motivation, feel free to make the argument a string, or the return value. But use those strings with care. If possible, use the proper type and avoid string, unless it leads to convoluted code.
Even if you don't use strings, you can end up with the wrong type. Let us motivate this with an example.
The Fibonacci sequence is defined as follows
\[ f_n = \begin{cases} 1 & n = 1\\ 1 & n = 2\\ f_{n-1} + f_{n-2} & \text{otherwise} \\ \end{cases} \]
Now, lets say you want a user to write a function that returns arbitrary Fibonacci numbers up to N = 100:
int32_t fibonacci(int8_t N);You might already note where I'm going with this, if not, read on. Lets have a look at \(f_{45}\), \(f_{46}\) and \(f_{47}\): \(1134903170, 1836311903, 2971215073\). And if you're familiar with the bounds of signed 32bit integer numbers, you should already see the problem.
The number \(2971215073\) cannot get represented with 32 bits (if signed integers are used), since \(\log_2 (2971215073) > 31\). If you were to ask the user for \(f_{50}\) (or even \(N = 100\)), you (and they!) end up with a wrong answer:
printf("f 47: %d\n", fibonacci(47));
// prints -1323752223This indicates that int32_t isn't enough for your kata. Even uint64_t isn't, since \(\log_2 (f_{100}) > 64\), so you have to switch to your languages BigInteger variant, e.g. BigInt, bignum, Integer.
By the way, this particular error could have been prevented by a test that checks that every returned number is positive. Also, if you use a dynamic typed language, make sure to check the return type in one of the first tests:
Test.expect (fib(0) is Bignum, "Your function doesn't return a BigNum")This is usually a mess. Your return type is correct, and you check for the correct type, but your users complain about invalid results. This indicates that you accidentally used the wrong type somewhere in your own computation, for example int16_t in a helper function. This is really hard to spot, so you should add some static test values. More on that in the later sections.
There's a later section on floating point numbers, but this section is also fitting for the delicate problem. As you (hopefully) know, floating point arithmetic isn't exact. The usual double value has 54 bits for its significant, which is better than int32_t, but worse than int64_t. So if you ask for an exact solution, you should ask for an integral type and also check that the user returns an integral type (in dynamic typed languages).
Note that JavaScript doesn't really differ between integral and floating point numbers. You can force numbers to behave as int32 (or uint32), but you cannot help the user with an appropriate error message in this case.
Also, JavaScript doesn't have a large integer class/type, so you need to improvise a little bit.
The preloaded code is both usable by you and the user. It's the perfect place to put helpers or data structures that the user should use, or to freeze objects in dynamic languages.
However, never put your solution into the preloaded code! Some languages provide reflection or similar means to check your preloaded code, which enables a user to cheat rather easily. Instead, put your solution into a local scope (see Hide your solution).
For more information about preloaded code, see the next section.
Sometimes, you'll ask your user to provide the answer in a given format, or to use certain pre-defined values. Those values should be preloaded too. For the format, either a fixed format string (printf style) or a full-fledged wonders helps you to get rid of a bunch of troubles, like additional white space, ambiguous syntax and so on.
Before we dive into tests in the next section, here's a word of warning. The next section does not replace the documentation of your testing framework. It gives general tips. However, in order to write good tests, a deep understanding of your testing framework is mandatory. Otherwise, it will take much time to write those, especially if you don't know the programming language well enough yet.
Have a look at the following test:
main = do
if solution 1 /= 1
then exitFailure
else
if solution 2 /= 1
then exitFailure
else
if solution 3 /= 2
then exitFailure
else
if solution 4 /= 3
then exitFailure
else
if solution 5 /= 3
then exitFailure
else return ()This test checks whether solution returns the first five Fibonacci numbers. It's bad for several reasons:
Another variant, which uses Hspec and random tests could be written as
main = do
rs <- replicateM 100 (randomRIO (1,100))
hspec $ do
describe "fibonacci" $ do
forM_ rs $ \x ->
it ("works for " ++ show x) $
fibonacci x `shouldBe` solution xWhile better, it doesn't use Hspec's interoperability with QuickCheck:
main = hspec $ describe "fibonacci" $ do
prop "works for random numbers" $ \x ->
fibonacci x `shouldBe` solution x
where
solution = ...So before you dive into the next section, make sure that you have at least a rough overview of the capabilities of your language's testing framework.
Now that you have read all those tips and tricks, you probably cannot wait to write your next kata, right? However, before you jump in, you should learn a little bit about the Codewars platform, if you want to run random tests or use the "preload" section.
Whenever you write a kata, you have at least six input windows:
All of this is used by the runner. The runner is basically a node process, which calls the languages interpreter and either
It then checks the output of the process for some fields, filters them, and shows the filtered text to the user. If the fields indicate that the user has passed the tests, they will have passed the kata.
This is the big difference between Codewars and other code running sites like SPOJ, Hackerrank and other. There, you will have to read from STDIN, and give formatted answers to STDOUT. You often don't learn details about your failed tests, only that you've failed them, since the tests are often only different *.txt files that get streamed into your file.
Unfortunately, the running parts of the Codewars runner are currently missing in the repository, but still available in its history. They are lacking new content like transpilers though.
In case of JavaScript, the code will get transformed via babel before it is passed to node. All of this preprocessing is not counted into the users runtime.
However, the important part is the concatenation in interpreted languages. That's the part where you have to look out, since it works like this:
$ cp python-framework run.py # add framework
$ echo "" >> run.py
$ cat preload.py >> run.py # add preloaded code
$ echo "" >> run.py
$ cat solution.py >> run.py # add user solution
$ echo "" >> run.py
$ cat tests.py >> run.py # add tests
$ cat run.py | python -e -The echo "" parts are there for newlines. Either way, as you can see, all of your and the user's code are in the same file. This means that while you have access on everything the user has defined, declared and written, it also means that they have access to everything you have written.
They can also change the test framework unless it's frozen. Keep in mind that there's an asynchronous cheating test, that adds non-passing tests. For example if the user solution exits prematurely with a "passed" output, they will get a penalty.
Note that there has been a heated discussion about the tests in Ruby. @danielpclark discusses some of the issues in the thread, and its worth a read.
Now to the other languages. For Haskell, the runner will automatically create a temporary dictionary structure that matches the module names. This dictionary will reside in /tmp/ and get run via runhaskell. The hspec framework has been replaced with another one that has some more features. Java and CSharp get handled similarly.
So remember: in interpreted languages, all your code and the user's code will reside in the same context, whereas in compiled languages all code will reside in the same directory.
By the way, this is the reason you cannot use process.argv in Node, as it would give a user immediate access to the tests.
The preloaded section in the kata editor is meant for things that should be done first, before any other code (beside the import of the test framework) is being read, evaluated or executed (in interpreted languages).
You want to provide some extra functions for the user? Put them in the preloaded section. You want to disable features, like Math.random or Math.abs? Do so in the preloaded section. You want to make sure that no one panders the testing framework? Freeze it in the preloaded section.
I think you get it. In interpreted languages, use the preloaded section for whatever helper you think the user might need, and for whatever security measures you want to enable. It's the first thing that gets read during the execution, so you're still in charge at that point.
In Haskell, the preloaded section needs a module name and can be imported. It's handy for additional functions or data types, as disabling features is done there with hidden.
Tests are the bread and butter of Codewars. Without tests, you couldn't check whether the user has solved your kata, so they're very important. Yet some neglect them.
Make sure to file an issue if you notice that a kata is lacking some tests. This will prevent a moderator from approving the kata. This will increase the overall quality of the approved katas. Also, as long as #123 stands, one cannot change the tests in approved katas with more than 500 solutions.
Before we actually dive into the many tips one can follow on Codewars, you should get familiar with the test driven development (TDD) cycle. It's not required to follow it in order to write katas for Codewars, but recommended.
So what is the cycle? Essentially, it's the following:
That's it. Note that the important part is the very first step: you write the tests first. "Why?", you're probably asking. "I know my solution already!" Well, are you sure about that? Some katas have long standing issues since the original author didn't check a few edge cases.
However, since Codewars isn't running on your own machine, you should adjust the cycle a little bit and add more than a single test. The cycle works best with fast iterations, and the remote nature of Codewars makes this a little bit hard. Still, writing at least part of your tests first is very recommended.
Let's have a look at the following test:
Test.expect(fizzBuzz(15) === 'FizzBuzz', "Value not what's expected")There are several things wrong with it. First of all, it uses expect. You should never use expect in any language, unless there isn't a fitting function in your test framework. But usually, there is a correct testing function. Here, one would use Test.assertEquals, but it might vary, depending on your language:
Test.assertEquals(fizzBuzz(15), 'FizzBuzz', "Value not what's expected")The additional message will be shown if the test failed. However, "Value not what's expected" is useless. It doesn't give the user any hint what's gone wrong:
Expected: "FizzBuzz", actual: "Buzz". Value not what's expected.Either drop the message in this case, or use a better one:
Test.assertEquals(fizzBuzz(15), 'FizzBuzz', "testing on 15")This tells the user immediately that their function failed on 15.
Furthermore, you should group your tests with describe and it. After all, a tests describes the behaviour of something, for example the behaviour of fizzBuzz. It should pass several specifications:
Test.describe('fizzBuzz', function(){
Test.it('should return "Fizz" on 3,6,9,12', function(){
for(let i = 1; i <= 4; ++i)
Test.assertEquals(fizzBuzz(3 * i), 'Fizz')
});
Test.it('should return "Buzz" on 5,10,20,25', function(){
for(let i = 1; i <= 4; i += 3)
Test.assertEquals(fizzBuzz(5 * i), 'Buzz')
Test.assertEquals(fizzBuzz(5 * (i + 1)), 'Buzz')
});
Test.it('should return "FizzBuzz" on 15*n', function(){
for(let i = 1; i <= 4; ++i)
Test.assertEquals(fizzBuzz(15 * i), 'FizzBuzz')
});
});Note that not every language provides describe and it. Refer to your language's test framework documentation to find out how to group tests there.
If you use describe, keep in mind that you describe one thing:
Test.describe('bar', function(){
Test.it('returns "bar"', function(){
Test.assertEquals(bar(), "bar");
});
});
Test.describe('foo', function(){
Test.it('returns "foo"', function(){
Test.assertEquals(foo(), "foo");
});
});A final remark on test grouping: if you have a look into some TDD books, you'll notice that they encourage you to use a single assertion in a test, not multiple as in this section's first example. This is also the subject of a programmers SE question.
However, some languages on Codewars don't provide the proper means to couple tests (label/text) with assertions, for example Python. It would be cumbersome to add a single it("...") for every assertion and clutters the test code. In those cases, you should write a helper that takes care of this.
Either way, you should always make sure that the user knows the reason for failing tests (see next section).
Especially if you use random tests (see below), you want to make sure that the user knows why the test failed. You could print the arguments. You could show a hint. Either way, a user shouldn't be left alone in face of an error.
Keep in mind that every output will be interpreted as HTML. If you want to use <, > or & in your error messages or description, make sure to escape it---unless you actually want to use HTML tags:
| Character | HTML entity | Mnemonic |
|---|---|---|
< |
< |
lesser than |
> |
> |
greater than |
& |
& |
ampersand |
All other characters are usually safe to use in UTF8 encoded documents.
If you ask the user to create a function that should work for 1 <= N <= 1,000,000, make sure that their function works for both 1 and 1,000,000. If you specified that "negative input should return a monkey", make sure that you actually test negative input.
If possible, try to use multiple kinds of assertions, e.g. assertEquals and assertNotEquals, or shouldSatisfy and shouldNotSatisfy. That way, a cheating user will have some more work to circumvent equality or predicate based tests.
Whenever when you state a kata you also have to write a solution. This is great, as you can use exactly that function to check the users code in the tests. What's even better: it makes you think about the input domain.
This is usually the hard part of creating random tests. After all, creating sufficient random input is most often harder than creating the kata itself. But it is worth every honour.
While it's nice to have a random test which validates that the user returns the same as your hidden solution, there's no guarantee that your solution provides the correct result. If you provide a predicate that checks whether the solution makes sense, you can use it in both the hidden and public tests.
For example, if foo should always return false on negative inputs, add the following test:
Test.it('should return false for negative numbers', function(){
for (let i = 0; i < 100; ++i) {
Test.assertEquals(foo(-Math.random() * 1000000), false);
}
});This prevents both you and the user to forget about those inputs. Generally this pattern will look like this:
Test.it('returns something valid', function(){
for (let i = 0; i < 100; ++i) {
var some = ...
var random = ...
var args = ...
Test.expect(
isValid(foo(some, random, args)),
"predicate failed, invalid answer returned"
);
}
});If you use random tests, make sure to hide your solution. In Java or C#, this includes making your function private. Haskell doesn't allow mutual imports, so the user cannot import the tests either way. However, in dynamic languages, one can sometimes simply use solve = solution.
You can prevent this kind of cheating if you
The first one is obvious. The second one can be realized if you don't provide a solution with the same interface, but instead a testAgainstSolution:
// bad!
var solution = function(a, b){
// ...
}
// better
var testFunction = function(a,b){
var solution = function(a, b) {
// ...
}
Test.assertEquals(userFunc(a,b), solution(a,b));
}There are more creative ways to hide/store the function, but that's one way at least. Note that all dynamic languages on Codewars (Ruby, Python, CS, JS) support this kind of local scopes.
Unless you're creating a puzzle where the user has to find the answer, present some example tests. Those tests should also contain the corner cases and some easy random tests, e.g. check that the user actually returns a number for arbitrary negative numbers.
Whenever the user returns a floating point number, make sure that you take floating point behaviour into account. The user might not add the numbers in the same order as you, and therefore end up with a slightly different number. Equality tests will always fail for that poor user.
Unless you want to write a kata were the user needs to round a floating point value to a certain precision, never ever tell the user to round up/down or truncate a floating point number. Instead, check the relative error. You might some time need to find a good epsilon, I usually use 1e-12 if I want to have something "almost exact" and 1e-7 if I want something "near-ish" the correct solution.
For more information about floating-point arithmetic, read What Every Computer Scientist Should Know About Floating-Point Arithmetic.
def add_three(a,b,c):
return a + b + c
def add_three_reference(a,b,c):
return a + (b + c)
print(add_three_reference(1e-12,1e-12,1) == add_three(1e-12,1e-12,1))
# FalseThe following example shows one way to check floating point values in a more sane way:
def assertFuzzyEquals(actual, expected, msg=""):
import math
# max error
merr = 1e-12
if expected == 0:
inrange = math.fabs(actual) <= merr
else:
inrange = math.fabs((actual - expected) / expected) <= merr
if msg == "":
msg = "Expected value near {:.12f}, but got {:.12f}"
msg = msg.format(expected, actual)
return Test.expect(inrange, msg)Most language sections contain their equivalent and use expect or a similar function of their test framework.
If your number is guaranteed to be an integral number, use int, long or your language's equivalent instead of floating point numbers. However, keep in mind that all values, either input, output, or intermediate should fit in this type, otherwise you might encounter overflow or underflow issues.
While #163 will give you the tools to fix a kata even late, you should take every reported test issue by heart. If the tests are failing for some only, check whether your current example tests contain enough corner cases. Ask them for their code (in spoilers).
There are several ways your tests can be wrong. For examples, your tests can be too strict. This is usually the case with floating points, where one expects the user to do things in a certain way.
However, sometimes your reference implementation – the solution the users try to write – is wrong. People have now adjusted their submitted functions to yours, so that they still solve the kata, even though their logic is broken. If you now fix your reference solution, the results of many users might get rejected.
So what can you do?
You should definitely tell your users if your kata is broken. Especially if your tests have a defect in only one particular language. It's not the best you can do, but as long as #163 is still there, it might be the only thing you can do.
This is fine as long as there haven't been many solutions. If your kata has been solved by hundreds, it might seem a little bit unfair, but correct tests are a little bit more important.
There are also other alternatives:
Either way, fix them, otherwise it will lead to submitted issues and therefore to negative honour.
The previous sections mentioned some practices for your tests. However, what if you don't know Haskell? Or Java? Or Ruby? You shouldn't provide tests for a language you don't know, as this will usually lead to bugs in your test cases (see section "A user solves a kata in a single language, not many").
That's where translations and collaboration kick in.
Let's stay in the mentioned scenario for a while. You wrote a kata, it's currently in beta, and suddenly, you get a notification that someone provided a translation. So where can you find that translation?
Go to your kata and have a look at the language icon bar. There's a "+" next to the language icons. Click on it to see the current list of translations. Here you get a quick overview of all translations. To see them in detail and approve, decline or fork them, click on the translation's title.
Now you can see their tests (both example and hidden), solution (both hidden and initial), and the description. There are several things you want to check before accepting a translation:
Those are basically questions that follow from the rules "Consistency is key" and "A user solves a kata in a single language, not many".
If all questions above can be answered with yes, you can approve the translation. If one or more questions were answered with no, write a comment on the translation. Discuss your concerns with the translator until both of you are happy. If you don't think that the translation is salvageable, decline it.
A translation basically copies the description at the time one has started writing the translation. If the kata's description changes before the translation got approved, the variants often cannot get merged together. This is called a merge conflict. (By the way, Codewars uses a 3-way merge).
Either the translator or the kata author has to fork the translation, fix the merge conflict by hand, and publish the translation again.
Now you found a kata in JavaScript. You liked it, and want to translate it into Python. There should be a "+" sign next to the language icons after you solved it. Click on it and you'll land on the list of current translations. After you've checked that there isn't a pending translation in Python, you click on "Translate this kata".
This will open translation kumite editor. Choose the language in the drop-down menu before you write any code. Then fill the (initial) solution and (example) tests. Save your progress often to prevent data loss. Whenever you're ready, publish your translation.
While the previous section included general tips, it's often not obvious how to implement them in a test framework. Random strings? Random numbers? How does one create specific elements?
This section gives some insights on those topics. However, it cannot replace your language's test framework documentation. You need to have at least some grasp of that one. Otherwise this section will probably confuse you.
CSharp is the only language that doesn't have a proper documentation on Codewars currently. This will probably change at some point. The test framework is NUnit, initially a port from JUnit. It's rather large, so we cannot cover everything here, but at least some points should be addressed.
Note: C# looks rather ugly in the generated PDF, so I'm going to refer to it as CSharp instead.
NUnit differs from the other frameworks. Very. Much. Well, overall it doesn't, it still has Assert.AreEqual, Assert.Greater and others. But the order of arguments is swapped.
Where CoffeeScript, Ruby, Python and Haskell expect the first argument of Test.assert_equals/Test.assertEquals/shouldBe to be the actual value and the second to be the expected, NUnit swaps the order, see the documentation of Assert:
Assert.AreEqual( int expected, int actual );
Assert.AreEqual( int expected, int actual, string message );
Assert.AreEqual( int expected, int actual, string message,
params object[] parms );
// many otherFor comparison, here's the syntax of an equality assert in the other languages:
-- Haskell
actual `shouldBe` expected// Javascript
Test.assertEquals(actual, expected, message);# Python
Test.assert_equals(actual, expected, message)# Ruby
Test.assert_equals(actual, expected, message)Keep this in mind. Your tests won't fail on correct solutions if you swap the arguments, but they will confuse anyone who fails them. By the way, Java uses the same convention. This isn't surprising, since NUnit was inspired by JUnit.
This was already addressed in the previous chapter, but it's rather important in CSharp. NUnit doesn't give you any method to label a test. Instead, a test is simply a public static method with a [Test] attribute inside a class with the [TestFixture] attribute. The name of your test is the method's name.
Make that name descriptive. A method called test1 doesn't help. Name it after the property you're testing, e.g. ReturnsTrueOnEven, or ThrowsExceptionOnNull.
[TestCase] to create parametrized testsA test function doesn't have to be of type void ...(void). You can use parameters, which will be shown automatically. This is possible via the TestCase attribute. It enables you to specify the arguments for a test function. This works best if you use TestCase together with Result:
[Test]
[TestCase(2, Result=true)]
[TestCase(4, Result=true)]
[TestCase(6, Result=true)]
[TestCase(-10, Result=true)]
public static bool ReturnsTrueOnEven(int n){
return Evener.isEven(n);
}
[Test]
[TestCase(3, Result=false)]
[TestCase(5, Result=false)]
[TestCase(7, Result=false)]
[TestCase(-1, Result=false)]
public static bool ReturnsFalseOnOdd(int n){
return Evener.isEven(n);
}Unfortunately, [TestCase] does not report the used parameters on a failed test. An alternative is to use Assert.* with a custom message:
[Test]
[TestCase(3, false)]
[TestCase(5, false)]
[TestCase(7, false)]
[TestCase(-1, false)]
public static void ReturnsFalseOnOdd(int n, bool result){
Assert.AreEqual(result, Evener.isEven(n),
String.Format("Returned the wrong result on {}", n));
}[Random] for simple random valuesThe Random attribute can be used on arguments. This enables you to create random integers or doubles within a range:
[Test]
public static void RandomTests([Random(10, 500000, 100)] int n)) {
Assert.AreEqual(PersistTests.Sol(n), Persist.Persistence(n));
}This would draw random integers between 10 and 500000 (inclusive) and run a total of 100 tests. Note that the drawbacks of [TestCase] also affect [Random], NUnit does not report the generated random values. Provide a message if you want your users to know what arguments they didn't get right.
[Theory] for a QuickCheck light experienceHaskell's QuickCheck enables you to test only certain combinations of random values via forAll. That's also possible with Theory and Assume:
[Theory]
public static void ReturnsTrueOnEven([Random(1, 50000, 100)] int n){
Assume.That(n % 2 == 0);
Assert.That(Evener.isEven(n));
}
[Theory]
public static void ReturnsFalseOnOdd([Random(1, 50000, 100)] int n){
Assume.That(n % 2 == 1);
Assert.That(! Evener.isEven(n));
}However, this should be used with care. After all, NUnit will still check only 100 random values, and if they don't hold the assumption, the test case is still completely executed but any failed assertion is disregarded. And yes, this will also not report the used values.
Haskell has been my main translation target on Codewars so far, so it's probably the one I know most. QuickCheck makes it tremendously easy to create random tests, and Hspec provides a very convenient test framework.
Codewars has QuickCheck, Hspec and HUnit installed. Unfortunately, it doesn't use a recent version of this libraries, but they're recent enough for most stuff you'll encounter 1.
Hspec's describe should be used with a function name, whereas it should be used with a sentence, so that it "should be readable". There can be many tests per it, but depending on your input you might want to use one test per it or a test by hand in accordance with expecationFailure instead of shouldBe. You can also use HUnit, but I usually stay within Hspec's DSL.
As written above, one shouldn't use shouldBe on floating point values, since the user might use another order of addition, and adding floating point numbers isn't associative. Therefore, you should plan a little threshold.
Hspec doesn't have such an operator, but you can easily write one yourself:
import Control.Monad (when)
shouldBeFuzzy :: (Fractional a, Ord a) => a -> a -> Expectation
shouldBeFuzzy a e =
when (abs ((a - e) / e) >= 1e-12) $ expectationFailure msg
where msg = "Expected " ++ show e ++ ", but got " ++ show aAlternatively, if you don't want to import Control.Monad, you can use
shouldBeFuzzy :: (Fractional a, Ord a) => a -> a -> Expectation
shouldBeFuzzy a e =
if cond
then expectationFailure msg
else return ()
where
msg = "Expected " ++ show a ++ ", but got " ++ show e
cond
| e == 0 = abs a >= 1e-12
| otherwise = (abs $ (a - e) / e) >= 1e-12forAll to constraint random valuesThe QuickCheck DSL is a little bit too large to explain completely, but usually that's not necessary. If you want to use an arbitrary Bool, Int or anything else from that list, simply do so while wrapping your test with property:
it "should work for arbitrary integers" $ property $ \x ->
2 * x `shouldBe` x * (2 :: Integer)However, the instance of Arbitrary must be clear. Therefore, you might need to add some types as can be seen above.
What if you want to restrict the random elements on some domain? That's when you use forAll or ==>. The first will generate only fitting values (if you use it correctly), the latter will discard values that don't fit:
it "should return true for even numbers" $ property $
forAll (arbitrary `suchThat` even) $ \x ->
solution x `shouldBe` True
it "should return true for even numbers" $ property $ \x -> even x ==>
solution x `shouldBe` TrueBeware! The latter variant might seem a lot easier, but if your domain is small, QuickCheck might give up. Note that this will also happen in the first case. If you want to generate a value, it's often better to generate an arbitrary value and then fmap, e.g. forAll (fmap (2*) arbitrary).
If you have several arguments which stem from different domains, use forAll per domain:
it "returns true on (3*n) (5*k)" $
property $
forAll (fmap (3*) arbitrary) $ \three ->
forAll (fmap (5*) arbitrary) $ \five ->
isFizzBuzz three five `shouldBe` TrueYou could also multiply the values later, but that will change the error message QuickCheck returns:
it "returns true on even values" $
forAll (arbitrary) $ \x ->
-- bad, prints `x`, not `2 * x`
isEven (2 * x) `shouldBe` True
it "returns true on even values" $
forAll (fmap (2*) arbitrary) $ \x ->
-- better, still prints x, but after
-- the application of (2*)
isEven x `shouldBe` TrueTo prevent name clashes, only import what you need from the user module. This enables the user to define additional helpers at global namespace.
Additionally, don't name your helper functions like QuickCheck, Hspec or HUnit operators or functions, even if they aren't currently available on Codewars. The administrators might update the package, and you end up with a defunct test due to ambiguity.
module Codewars.MyPrefix.KataName.Test where
import Codewars.MyPrefix.KataName (functionName1, functionName2, ...)
import Test.Hspec
import Test.QuickCheck
-- Solving modules, e.g. Data.List, Data.Char, sometimes qualified:
import Module.That.Solves.The.Kata
import Text.Printf (printf) -- if you need to use printf
main = hspec $ do
describe "functionName1" $ do
-- multiple tests per it; only reports expected vs actual
it "should work for some simple examples" $ do
functionName1 "abc" `shouldBe` "abc"
functionName1 "xyz" `shouldBe` "xyz"
functionName1 "" `shouldBe` "" -- corner case
-- multiple tests per it; hand written failure message
it "should work for some other examples" $ do
let test i r = if a /= r
then expectationFailure $ p
else return ()
where
a = functionName1 i
p = concat ["expected ", show r, " on "
, show i, " but got ", show a
]
test "def" "def"
test "123" ""
it "shouldn't change letter-only strings" $ property $
forAll (listOf $ elements $ ['a'..'z'] ++ ['A'..'Z']) $ \xs ->
functionName1 xs `shouldBe` xs
it "should return the right result" $ property $ \xs ->
functionName1 xs `shouldBe` solution1 xs
describe "functionName2" $ do
-- ... similar to aboveCodewars' Java test framework uses JUnit version 4.11. The latest version at this point is 4.12, so make sure to check the framework's documentation for recent changes if you encounter problems. The functionality you use might not be in JUnit 4.11.
Note: This section is under construction and could use some help. Please submit pull-requests.
JUnit differs from the other frameworks. Very. Much. Well, overall it doesn't, it still has assertEquals, assertArrayEquals and others. But the order of arguments is swapped.
Where CoffeeScript, Ruby, Python and Haskell expect the first argument of Test.assert_equals/Test.assertEquals/shouldBe to be the actual value and the second to be the expected, JUnit swaps the order, see the documentation of Assert:
assertEquals(double expected, double actual); // note: use delta variant!
assertEquals(float expected, float actual); // note: use delta variant!
assertEquals(long expected, long actual);
// many otherFor comparison, here's the syntax of an equality assert in the other languages:
-- Haskell
actual `shouldBe` expected// Javascript
Test.assertEquals(actual, expected, message);# Python
Test.assert_equals(actual, expected, message)# Ruby
Test.assert_equals(actual, expected, message)Keep this in mind. Your tests won't fail on correct solutions if you swap the arguments, but they will confuse anyone who fails them. By the way, CSharp uses the same convention. This isn't surprising, since its testing framework NUnit was inspired by JUnit.
The JavaScript tests on Codewars use a custom test framework. Any kata written in CoffeeScript will use the same framework, since CoffeeScript gets compiled to JavaScript.
Therefore, you won't find a section about CoffeeScript in this document. Also, keep in mind that the current support of ES2015 is implemented via Babel, so you won't get the real source code if you use .toString(), but the result of the Babel compiler.
As written above, one shouldn't use assertEquals on floating point values, since the user might use another order of addition, and adding floating point numbers isn't associative:
console.log("Is + associative in JS on double values?")
console.log(1 + (1e-12 + 3e-12) === (1 + 1e-12) + 3e-12)
console.log(" :( ")Therefore, you should plan a little threshold.
ES6 provides Number.EPSILON, but that's actually a little bit too low for most cases. I usually use 1e-12, which is good enough, or 1e-6 if the kata is about approximative results. Just change the value below to whatever you feel is best. Just keep in mind that any value lower than Math.min(1e-15,Number.EPSILON) is bogus.
var assertFuzzyEquals = function(actual, expected, msg){
var inrange = Math.abs((actual - expected) / expected) <= 1e-12;
Test.expect(expected !== 0 ? inrange : Math.abs(actual) <= 1e-12,
msg || "Expected value near " + expected.toExponential(13) +
", but got " + actual.toExponential(13)
);
}@slothpie provided the following code to check the user's code for forbidden functions:
Test.describe("Reinforcement Test Cases:", function() {
// get the code as text, (this also grabs test case code)
var dump = arguments.callee.caller.caller.caller.toString();
// The length of our test case code.
const FIXED = arguments.callee.caller.caller.toString().length;
// Slice out our test case code so we just have users
const USER_CODE = dump.slice(0,dump.length-FIXED);
check_user_code_for_stuff(USER_CODE);
});However, there are two things you have to keep in mind:
USER_CODE does not contain actual user code, but the one returned by babel.js. You never get access to the actual source code (until node supports ES6 and gets updated on Codwars).arguments.callee and arguments.caller only work if you don't run your script in strict mode. A stray "use strict"; in the user's code will lead to an error, even if the code is otherwise OK.Other than that, it's an easy way to check whether the user uses any loops or similar.
In August 2015 the node-quickeck package was added to the available JavaScript packages. This section predates the inclusion of the package. If you want to use node-quickcheck, have a look at it's documentation. If you prefer to write your random tests by hand, read on.
The following functions can be used for random tests. Note that both expect your solution to be pure (don't change the argument, for example if it's an object or an array) and cannot check side-effects, so you probably edit the functions for your needs. Also note that randomAssertSimilar is rather verbose.
In both functions, generator should return an array.
NOTE: The functions haven't been tested thoroughly. Use them with care and feel free to create an issue or a pull-request if they contain any errors.
/**
* @brief Tests a user defined function against a reference function.
* @param generator generates random arguments (must return an array)
* @param userSol is the solution provided by the user
* @param refSol is the solution provided by the original kata author
* @param tests is the number of tests (optional)
*
* The `refSol` should be pure, it shouldn't modify it's arguments,
* since `userSol` will work on the same array afterwards.
*
* The values are compared via equality (===), and the number of
* shown tests is limited to three.
*/
var randomAssertEquals = function(generator, userSol, refSol, tests){
tests = tests || 100;
var i = 0, user, reference, values;
while( tests --> 0){
values = generator();
reference = refSol.apply(this, values);
user = userSol.apply(this, values);
if(i++ < 3){
Test.assertEquals(
user, reference,
"didn't work on the following argument array: " + values
);
} else if(reference !== user){
Test.assertEquals(
user, reference,
"didn't work on the following argument array: " + values
);
}
}
}/**
* @brief Tests a user defined function against a reference function.
* @param generator generates random arguments (must return an array)
* @param userSol is the solution provided by the user
* @param refSol is the solution provided by the original kata author
* @param tests is the number of tests (optional)
*
* The `refSol` should be pure, it shouldn't modify it's arguments,
* since `userSol` will work on the same array afterwards.
*
* The values are compared via assertSimilar.
*/
var randomAssertSimilar = function(generator, userSol, refSol, tests){
tests = tests || 100;
var user, reference, values;
while( tests --> 0){
values = generator();
reference = refSol.apply(this, values);
user = userSol.apply(this, values);
Test.assertSimilar(
user, reference,
"didn't work on the following argument array: " + values
);
}
}Test.describe("functionName1", function(){
var solution = function(){
// your solution
};
Test.it("should work for some simple examples", () => {
Test.assertEquals(functionName1("abc"), "abc");
Test.assertEquals(functionName1("xyz"), "xyz");
Test.assertEquals(functionName1("") , ""); // corner case
});
Test.it("should not return strings longer than 3 chars", () => {
for(let i = 0; i < 100; ++i){
const str = Test.randomToken();
Test.expect(functionName1(str).length <= 3,
'returned more than 3 characters on' + str);
}
});
Test.it("returns the correct string", () => {
for(let i = 0; i < 100; ++i){
const str = Test.randomToken();
Test.assertEquals(
functionName1(str),
solution(str)
);
}
});
});Sorry. This section does not contain any helpful information currently. Instead, have a look at the following PEPs:
If you have some ideas for this section, head over to GitHub.
As written above, one shouldn't use assert_equals on floating point values, since the user might use another order of addition, and adding floating point numbers isn't associative:
print ("Is + associative in Python on double values?")
print (1 + (1e-12 + 3e-12) == (1 + 1e-12) + 3e-12)
print (" :( ")Therefore, you should plan a little threshold. I usually use 1e-12, which is most often good enough, or 1e-6 if the kata is about approximative results. Just change the value below to whatever you feel is best. Keep in mind that any value lower than 1e-15 is bogus.
def assertFuzzyEquals(actual, expected, msg=""):
import math
# max error
merr = 1e-12
if expected == 0:
inrange = math.fabs(actual) <= merr
else:
inrange = math.fabs((actual - expected) / expected) <= merr
if msg == "":
msg = "Expected value near {:.12f}, but got {:.12f}"
msg = msg.format(expected, actual)
return Test.expect(inrange, msg)Note: This section is under construction and could use some help. Please submit pull-requests.
Codewars uses Ruby 2.0. The testing framework is inspired by RSpec, but not 100% compatible.
This section answers some questions which are not related to the topics in the first chapters, but asked rather often on Codewars. Compared to the rest of the document, this section shouldn't get read in a linear style. Instead, just check whether your question is here whenever you have one in mind. If it isn't, file an issue on GitHub.
Nothing is worse on Codewars than writing a solution and getting an error that isn't explained. However, sometimes the error doesn't stem from the kata, but from the Codewars system itself.
This section is dedicated to those errors.
Once in a while, you can encounter the following error:
Submission timed out. Please try again.
First of all, don't worry. This doesn't mean that your code is wrong. However, it doesn't mean that your code is correct either. Instead, your code wasn't even tested. The very process of submitting the code timed out. This can happen under heavy load, if all worker pools are exhausted, all queues are full, or some other behind the scenes detail gave up.
The actual source of the problem isn't fully known yet. Sometimes, it's background jobs not processing items from the job queue. Sometimes, it's another, completely unrelated process on the same machine, that takes too much resources from a Codewars virtual machine.
First of all, relax. Even if the submission timed out, your code is usually saved. However, you can copy your code into your clipboard and reload the kata submission page in order to check this: if the code is still there, you can come back later. If it isn't, paste it and try to submit it again – maybe it was just a lost server connection and you can go on.
If the problem persists you can notify Jake or at least report the incident in the aforementioned issue topic (#18). You can also check the status page. Note that the status page isn't always accurate. Also have a look at the dashboard. People usually post issues or comments about "submission timed out" if this happens.
Again, this error doesn't solely happen to you, and it doesn't have anything to do with your code. Please do not comment on katas in this case, as "Submission timed out" comments clutter the comment page, and the kata author can't do anything to fix this error.
That, on the other hand, is a code related error. It means that your code wasn't able to perform the necessary operations in the given time. The time limit varies between languages, but it's usually ~6s. Therefore, this is an error you can fix. You need to perform optimizations on your code. This includes memoization, dynamic programming or even better algorithms and depends on the kata. If you get stuck, submit a comment on the specific kata and mark it as "Question".
Sure.
| Error message | Interpretation / meaning |
|---|---|
| Submission timed out | The Codewars platform has some problems. Try again later. Your code wasn't checked. |
| Process timed out | Your code is to slow and has exceeded the time limit. Try another approach, algorithm, or optimize. |
Sometimes, you will encounter an "Unknown error". The complete error message is
Tests didn't pass: Unknown error.
However, this error message is usually misleading. Most important: the error isn't affiliated with your code. Instead, the test framework didn't actually run.
You will also encounter the error on empty tests. Some katas don't provide example test cases. Run won't work in those circumstances, since there are no tests to run. Instead, press Submit on those katas.
Another workaround that works sometimes is submit -> reload -> submit. However, this doesn't work all the time.
Apparently, the libraries do get updated from time to time, but there's no reference at the moment.↩