Help needed to verify a new on-line C test

Do you have 2 years or more experience of C programming?

I am looking for 50 volunteers to help verify the reliability of a new on-line C test.

Each test consists of 20 multiple-choice questions which can be answered in 30 minutes.

Each participant should take two tests, ideally on consecutive days.

As a "thank you" you will get 10 free test credits on Oxford Score - On-line testing ($100 value). However, note that the contents of the site is under development and you may not be able to use your credits until the content is finalised.

To participate email [email protected] and you will get your test invitation by email. To see what the tests look like, take one of the free tests on Tests available

Many thanks.

> Do you have 2 years or more experience of C programming?
Does that matter?

Because I've met people who claimed far more years, yet were basically still ignorant.

What are "test credits" and why would I want them?

Quote:

---

Originally Posted by Salem

> Do you have 2 years or more experience of C programming?
Does that matter?

Because I've met people who claimed far more years, yet were basically still ignorant.

---

Sure. 2 years of experience is not a guarantee of proficiency, but it does increase the probability that the person is proficient.

Quote:

---

Originally Posted by robwhit

What are "test credits" and why would I want them?

---

See Oxford Score - On-line testing

Forgive me for being cynical, but I'm so doubtful about the accuracy of C testing. Sometimes I wonder if the people giving tests know what they're doing.

But yeah, 2 years is not really a long enough time to be proficient at C, at least not for most people that I seem to come across.

Plus I don't think testing syntax really shows much, although I'm sure the questions are more than just that :-).

Instead of asking for 50 volunteers, which is really a fair bit, why not just release the questions to the public? Or are you afraid that could damage the test in the future?

Quote:

---

Originally Posted by MacGyver

Forgive me for being cynical, but I'm so doubtful about the accuracy of C testing. Sometimes I wonder if the people giving tests know what they're doing.

---

I fully agree that bad tests are not likely to produce valid results. However, I think that I have produced a fairly good test and the statistical analysis which I am undertaking will verify, amongst other things, its validity.

Why not try it?

Quote:

---

Originally Posted by zacs7

Plus I don't think testing syntax really shows much, although I'm sure the questions are more than just that :-).

Instead of asking for 50 volunteers, which is really a fair bit, why not just release the questions to the public? Or are you afraid that could damage the test in the future?

---

Ideally, I would prefer 100 or more since the statistical validity of the findings increases with the number.

On your last point, yes, your conclusion is correct.

Well I did demo test 1.

Some of the code could be better formatted (or at least with better visibility). I seemed to spend a lot of time just trying to read the code rather than working out what it did.

A consistent formatting style to the code as well would help. Code formatted in unusual ways takes more effort (and time) in just figuring out what it does, before the comprehension has to kick in.

The ticking time is really annoying, and I'm not sure what it proves.

Is say 15 out of 30 in 30 minutes indicative of a better programmer, than one who could get 30 out of 30 in say 60 minutes. Personally (in the work-place), I would much rather value the latter approach. Being fast, but wrong doesn't help anybody.

Take the number of right answers and fit that to one bell-curve, and take the time taken for all the questions and fit that to another bell-curve and produce your composite answer.
With a sufficiently large sample set, you'll have a good idea of how long the test SHOULD take for a competent person, and not the random number of minutes you decided it ought to take.

Measure the time taken, just don't sit there with a stopwatch asking "are you done yet every single second". Sure, if someone is nowhere near finished, and they're past the upper time limit of say 95% of responders, then you might call a halt to the test.

The "do not use a compiler" is a complete red-herring. The effort to get just a few questions up and running would not give the answer in most cases, and burn up a hell of a lot of time in the process. Particularly where the answer is "undefined", you also list the probable answers that common compilers routinely produce as "answers", you'll soon be able to spot those with fast fingers who tried to compile it (they picked a common, but mis-leading answer direct from the compiler), from those who actually know the answer is "undefined".

Thank you for doing the sample test and your subsequent comments.

As you know, the formatting question is a matter of taste and most probably company coding standards. In my experience a good programmer will (and should) be able to read code in most reasonable formatting conventions (but always in a non-proportional font): hence the variety in the questions.

Re the clock countdown annoying you: I felt it was helpful to know how much time you have left, but as it could be annoying to some, I will add the "kill the clock" button.

I found that some of the questions had a bit of a dubious answer: For example, a program that accesses one element outside the defined range of an array may not crash - it MAY crash, but it also may not.

I didn't do as well as I expected - not sure if that's because I got things wrong because I was quick (I didn't take even close to the full time for either of my two attempts). I got 67% and 81% - I guess that's better than average, but I expected that I'd get better than that, since I'm generally quite good at understanding and spotting things that are wrong. I guess I belong to the category that Salem wouldn't want to work with - fast but not always right ;)

I also think that a relatively large amount of the questions where dealing with undefined behaviour or bad coding - whilst it is important to understand these things, it is not necessarily the ONLY important thing to understand, and a more generic "can you read code" type questions would perhaps make more sense.

--
Mats

> For example, a program that accesses one element outside the defined range of an array
> may not crash - it MAY crash, but it also may not.
Well that sums up all undefined behaviour. Even if it doesn't crash immediately, it may have triggered a chain of events which would lead to a crash.
I suppose the tone of that question (I didn't get that one on my random set) would be to see if you knew you could POINT to the N+1 element of the array for comparison with another pointer.

Quote:

---

Originally Posted by matsp

I found that some of the questions had a bit of a dubious answer: For example, a program that accesses one element outside the defined range of an array may not crash - it MAY crash, but it also may not.

I didn't do as well as I expected - not sure if that's because I got things wrong because I was quick (I didn't take even close to the full time for either of my two attempts). I got 67% and 81% - I guess that's better than average, but I expected that I'd get better than that, since I'm generally quite good at understanding and spotting things that are wrong. I guess I belong to the category that Salem wouldn't want to work with - fast but not always right ;)

I also think that a relatively large amount of the questions where dealing with undefined behaviour or bad coding - whilst it is important to understand these things, it is not necessarily the ONLY important thing to understand, and a more generic "can you read code" type questions would perhaps make more sense.

--
Mats

---

Thank you for doing the test and your comments. You did very well on a tough test. The average for a large (100+) pool of job applicants on a very similar test would typically be around 35%.

One of the reasons for the "toughness" is that points are deducted for wrong answers in order to discourage guessing. You do not loose points for clicking "pass".

Thank you also for your suggestion for more "can you read code" questions: this has been noted for future development.

Quote:

---

Originally Posted by JanHruska

Thank you for doing the test and your comments. You did very well on a tough test. The average for a large (100+) pool of job applicants on a very similar test would typically be around 35%.

---

But how many of those have 15+ years experience of programming in C and C++?

--
Mats

Quote:

---

Originally Posted by matsp

But how many of those have 15+ years experience of programming in C and C++?

---

35% was the average score for the original sample. Unfortunately, I do not recall the standard deviation, which would paint a better picture. A part of the test verification that I am doing is to find out the stats for this test.

My estimate is that IRO 1 in 20 applicants in the original sample had 15+ years of C/C++.