Thread: Help needed to verify a new on-line C test

  1. #1
    Registered User JanHruska's Avatar
    Join Date
    Jun 2009
    Location
    Oxford, England
    Posts
    8

    Help needed to verify a new on-line C test

    Do you have 2 years or more experience of C programming?

    I am looking for 50 volunteers to help verify the reliability of a new on-line C test.

    Each test consists of 20 multiple-choice questions which can be answered in 30 minutes.

    Each participant should take two tests, ideally on consecutive days.

    As a "thank you" you will get 10 free test credits on Oxford Score - On-line testing ($100 value). However, note that the contents of the site is under development and you may not be able to use your credits until the content is finalised.

    To participate email [email protected] and you will get your test invitation by email. To see what the tests look like, take one of the free tests on Tests available

    Many thanks.

  2. #2
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,656
    > Do you have 2 years or more experience of C programming?
    Does that matter?

    Because I've met people who claimed far more years, yet were basically still ignorant.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  3. #3
    Registered User
    Join Date
    Oct 2001
    Posts
    2,129
    What are "test credits" and why would I want them?

  4. #4
    Registered User JanHruska's Avatar
    Join Date
    Jun 2009
    Location
    Oxford, England
    Posts
    8
    Quote Originally Posted by Salem View Post
    > Do you have 2 years or more experience of C programming?
    Does that matter?

    Because I've met people who claimed far more years, yet were basically still ignorant.
    Sure. 2 years of experience is not a guarantee of proficiency, but it does increase the probability that the person is proficient.

  5. #5
    Registered User JanHruska's Avatar
    Join Date
    Jun 2009
    Location
    Oxford, England
    Posts
    8
    Quote Originally Posted by robwhit View Post
    What are "test credits" and why would I want them?
    See Oxford Score - On-line testing

  6. #6
    Deathray Engineer MacGyver's Avatar
    Join Date
    Mar 2007
    Posts
    3,210
    Forgive me for being cynical, but I'm so doubtful about the accuracy of C testing. Sometimes I wonder if the people giving tests know what they're doing.

    But yeah, 2 years is not really a long enough time to be proficient at C, at least not for most people that I seem to come across.

  7. #7
    Woof, woof! zacs7's Avatar
    Join Date
    Mar 2007
    Location
    Australia
    Posts
    3,459
    Plus I don't think testing syntax really shows much, although I'm sure the questions are more than just that :-).

    Instead of asking for 50 volunteers, which is really a fair bit, why not just release the questions to the public? Or are you afraid that could damage the test in the future?
    Last edited by zacs7; 06-16-2009 at 04:20 AM.

  8. #8
    Registered User JanHruska's Avatar
    Join Date
    Jun 2009
    Location
    Oxford, England
    Posts
    8
    Quote Originally Posted by MacGyver View Post
    Forgive me for being cynical, but I'm so doubtful about the accuracy of C testing. Sometimes I wonder if the people giving tests know what they're doing.
    I fully agree that bad tests are not likely to produce valid results. However, I think that I have produced a fairly good test and the statistical analysis which I am undertaking will verify, amongst other things, its validity.

    Why not try it?

  9. #9
    Registered User JanHruska's Avatar
    Join Date
    Jun 2009
    Location
    Oxford, England
    Posts
    8
    Quote Originally Posted by zacs7 View Post
    Plus I don't think testing syntax really shows much, although I'm sure the questions are more than just that :-).

    Instead of asking for 50 volunteers, which is really a fair bit, why not just release the questions to the public? Or are you afraid that could damage the test in the future?
    Ideally, I would prefer 100 or more since the statistical validity of the findings increases with the number.

    On your last point, yes, your conclusion is correct.

  10. #10
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,656
    Well I did demo test 1.

    Some of the code could be better formatted (or at least with better visibility). I seemed to spend a lot of time just trying to read the code rather than working out what it did.

    A consistent formatting style to the code as well would help. Code formatted in unusual ways takes more effort (and time) in just figuring out what it does, before the comprehension has to kick in.

    The ticking time is really annoying, and I'm not sure what it proves.

    Is say 15 out of 30 in 30 minutes indicative of a better programmer, than one who could get 30 out of 30 in say 60 minutes. Personally (in the work-place), I would much rather value the latter approach. Being fast, but wrong doesn't help anybody.

    Take the number of right answers and fit that to one bell-curve, and take the time taken for all the questions and fit that to another bell-curve and produce your composite answer.
    With a sufficiently large sample set, you'll have a good idea of how long the test SHOULD take for a competent person, and not the random number of minutes you decided it ought to take.

    Measure the time taken, just don't sit there with a stopwatch asking "are you done yet every single second". Sure, if someone is nowhere near finished, and they're past the upper time limit of say 95% of responders, then you might call a halt to the test.

    The "do not use a compiler" is a complete red-herring. The effort to get just a few questions up and running would not give the answer in most cases, and burn up a hell of a lot of time in the process. Particularly where the answer is "undefined", you also list the probable answers that common compilers routinely produce as "answers", you'll soon be able to spot those with fast fingers who tried to compile it (they picked a common, but mis-leading answer direct from the compiler), from those who actually know the answer is "undefined".
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  11. #11
    Registered User JanHruska's Avatar
    Join Date
    Jun 2009
    Location
    Oxford, England
    Posts
    8
    Thank you for doing the sample test and your subsequent comments.

    As you know, the formatting question is a matter of taste and most probably company coding standards. In my experience a good programmer will (and should) be able to read code in most reasonable formatting conventions (but always in a non-proportional font): hence the variety in the questions.

    Re the clock countdown annoying you: I felt it was helpful to know how much time you have left, but as it could be annoying to some, I will add the "kill the clock" button.

  12. #12
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    I found that some of the questions had a bit of a dubious answer: For example, a program that accesses one element outside the defined range of an array may not crash - it MAY crash, but it also may not.

    I didn't do as well as I expected - not sure if that's because I got things wrong because I was quick (I didn't take even close to the full time for either of my two attempts). I got 67% and 81% - I guess that's better than average, but I expected that I'd get better than that, since I'm generally quite good at understanding and spotting things that are wrong. I guess I belong to the category that Salem wouldn't want to work with - fast but not always right

    I also think that a relatively large amount of the questions where dealing with undefined behaviour or bad coding - whilst it is important to understand these things, it is not necessarily the ONLY important thing to understand, and a more generic "can you read code" type questions would perhaps make more sense.

    --
    Mats
    Last edited by matsp; 06-18-2009 at 05:30 PM.
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  13. #13
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,656
    > For example, a program that accesses one element outside the defined range of an array
    > may not crash - it MAY crash, but it also may not.
    Well that sums up all undefined behaviour. Even if it doesn't crash immediately, it may have triggered a chain of events which would lead to a crash.
    I suppose the tone of that question (I didn't get that one on my random set) would be to see if you knew you could POINT to the N+1 element of the array for comparison with another pointer.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  14. #14
    Registered User JanHruska's Avatar
    Join Date
    Jun 2009
    Location
    Oxford, England
    Posts
    8
    Quote Originally Posted by matsp View Post
    I found that some of the questions had a bit of a dubious answer: For example, a program that accesses one element outside the defined range of an array may not crash - it MAY crash, but it also may not.

    I didn't do as well as I expected - not sure if that's because I got things wrong because I was quick (I didn't take even close to the full time for either of my two attempts). I got 67% and 81% - I guess that's better than average, but I expected that I'd get better than that, since I'm generally quite good at understanding and spotting things that are wrong. I guess I belong to the category that Salem wouldn't want to work with - fast but not always right

    I also think that a relatively large amount of the questions where dealing with undefined behaviour or bad coding - whilst it is important to understand these things, it is not necessarily the ONLY important thing to understand, and a more generic "can you read code" type questions would perhaps make more sense.

    --
    Mats
    Thank you for doing the test and your comments. You did very well on a tough test. The average for a large (100+) pool of job applicants on a very similar test would typically be around 35%.

    One of the reasons for the "toughness" is that points are deducted for wrong answers in order to discourage guessing. You do not loose points for clicking "pass".

    Thank you also for your suggestion for more "can you read code" questions: this has been noted for future development.

  15. #15
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Quote Originally Posted by JanHruska View Post
    Thank you for doing the test and your comments. You did very well on a tough test. The average for a large (100+) pool of job applicants on a very similar test would typically be around 35%.
    But how many of those have 15+ years experience of programming in C and C++?

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Pointer and Polymorphism help.
    By Skyy in forum C++ Programming
    Replies: 29
    Last Post: 12-18-2008, 09:17 PM
  2. print line by line from a file
    By SoFarAway in forum C Programming
    Replies: 3
    Last Post: 02-18-2005, 01:36 PM
  3. Read only one line using seekg
    By RedZippo in forum C++ Programming
    Replies: 3
    Last Post: 03-31-2004, 11:10 PM
  4. if is faster than switch?
    By skorman00 in forum C++ Programming
    Replies: 32
    Last Post: 03-06-2004, 01:15 PM