Forum: support
Monitor Forum | Start New ThreadRE: notes about automatic recognition of tests [ Reply ] By: Achim Zeileis on 2018-07-30 12:58 | [forum:46181] |
OK, great, thanks for the info. Re: xelatex. If you end up recommending a solution based on xelatex, it would be good to provide some more information about the advantages and disadvantages of pdflatex vs. xelatex. The latter would introduce a new system requirement and the question would be whether this is worth the effort. In my very limited personal experience the wncyss-based solution works quite well and looks reasonably nice. But you can surely judge this better than I can. |
RE: notes about automatic recognition of tests [ Reply ] By: Boris Demeshev on 2018-07-30 00:52 | [forum:46179] |
Not yet, I will certainly play with fonts and xelatex engine later. I think it will happen in late August or September. |
RE: notes about automatic recognition of tests [ Reply ] By: Achim Zeileis on 2018-07-30 00:30 | [forum:46177] |
Boris, did you have a chance to look at this? Do you have any guidance/feedback regarding the cyrillic sans-serif language/font/encoding? |
RE: notes about automatic recognition of tests [ Reply ] By: Achim Zeileis on 2018-07-06 23:43 | [forum:46133] |
I have just commited an update of the "exams" package to the SVN which now includes the ru.dcf file (thanks again) and also ensures that type/id/scrambling are in Helvetica {phv} so that these can be properly scanned. The following should now work: exams2nops(..., header = "\\usepackage[russian]{babel}", language = "ru") But using a sans serif font would probably be more consistent with the exams2nops design. So one could do: exams2nops(..., header = "\\usepackage[OT2]{fontenc}\n\\usepackage[russian]{babel}\n\\renewcommand{\\rmdefault}{wncyss}\n\\renewcommand{\\sfdefault}{wncyss}", language = "ru") As you have probably much more experience with this: Would the latter be ok? Is there a more elegant way to do this or something similar? When I have a better understanding of how to best set language/font/encoding, I will also wire this properly into exams2nops(). (But before I'll be on vacation for the next week... :-)) Thanks in advance! |
RE: notes about automatic recognition of tests [ Reply ] By: Achim Zeileis on 2018-06-30 00:25 | [forum:46111] |
OK, I understand! I have tweaked the header argument to also process header = "\\usepackage[russian]{babel}" This has been committed to the SVN which should rebuild the package shortly. I also discovered why you had problems scanning the exam ID! Adding the Russian babel changes the font which is not good for the optimal character recognition. This depends on the Helvetica (phv) font being used. One solution would be to enforce phv being used for the ID/Scrambling/Type. Alternatively: Is it possibly to use some sort of sans serif font (preferably like Helvetica) along with the Russian babel package? |
RE: notes about automatic recognition of tests [ Reply ] By: Boris Demeshev on 2018-06-29 15:13 | [forum:46108] |
Let's continue with Russian localization :) I need \usepackage[russian]{babel} in tex or better \input{add_in.tex} above all possible russian letters. * usepackage = "[russian]{babel}" will produce \usepackage{[russian]{babel}} * header = "\usepackage[russian]{babel}" will not work as \u is special character * header = "\\usepackage[russian]{babel}" will produce \{\usepackage[russian]{babel}} in latex output * intro = "\usepackage[russian]{babel}" will not work as \u is a special character * intro = "\\usepackage[russian]{babel}" will work (!!! compare to header option!!!) and produce expected \usepackage[russian]{babel} but it will be too late :) Cyrillic letters are above intro! |
RE: notes about automatic recognition of tests [ Reply ] By: Achim Zeileis on 2018-06-29 14:13 | [forum:46107] |
Boris, wonderful, thanks for the ru.dcf and for the feedback. This is much appreciated. Some disclaimer first: R/exams has been developed for my own exams and then got picked up by others at our department/faculty etc. Because I think that it can be useful to others, I write documentation and provide support. But given that it is something that I do "on the side" I just didn't get round to implementing all the ideas I have for improving it. The same holds for the other contributors to the package. Hence, we are well aware of some of the potential pitfalls you mention but didn't manage to implement a good solution, yet. As for the notes: 1. You can include LaTeX code with "usepackage", "header", and "intro". In your case the latter would probably be the simplest solution. But I haven't tried it yet, will do so another time. 2. I'm not assuming that life is smooth - but for our exams I put a lot of effort into making the exam sheets as "clean" as possible. Hence, for me, it is usually just nops_scan + nops_eval without additional manual intervation. But, of course, I have intervened manually more than a few times. So I know exactly that this was not easy/pleasant for you. I do have various ideas for your step B and a lot of the building blocks are available in the package - tying them properly together is not trivial though. 3. This is a problem that I have seen only very rarely. Maybe the resolution of your JPG scans is too poor? If you can post a dummy scanned sheet (i.e., without "real" student information), I could have a look. Additionally, it may be worth to check whether the options of your scanner can be tweaked. This typically pays off in the end :-) Our department just bought a new scanner and I spent a few hours experimenting with gray vs. b/w scanning and different resolutions, formats, gray levels, etc. As for the improvement suggestions: 1./2. It is unlikely that nops_scan will gain many of the features that you suggest. The main reason is that nops_scan itself should run without user interaction and without any further information about the exam and/or the participants, possibly on a remote multi-core system. However, better tools for reading/processing the resulting output would be useful, yes. 3. This is exactly the direction in which I was thinking but didn't implement anything yet. There should be some validator that tries to resolve problems _while_ problems still exist. (Note that fixing some problems might introduce new/other problems.) Also, I would like to have a function where I can simply enter all parts of the sheet manually, i.e., use my own eyes for scanning. 4. The text-based format is needed for compatibility with a system that is used at various Austrian universities. Hence also the German file name "Daten.txt". 5. This is not an issue I have encountered. I simply process the scans file-by-file. Hence, I didn't have any memory problems and could easily parallelize the code with mclapply. In any case, thanks for all the feedback. I would suggest to: (a) Continue this thread for resolving the adaptation to Russian. (b) Open another thread for the issues around handling the output from your scanner. (c) Possibly start a thread for validation. But for this the coding will surely take longer. Another disclaimer: I will be traveling to a conference and then family vacation, starting from Sunday. So please bear with me if I don't reply very timely... Thanks again! |
notes about automatic recognition of tests [ Reply ] By: Boris Demeshev on 2018-06-29 11:12 | [forum:46106] ru.dcf (3) downloads |
Ten days ago we have tried doing test for 230 students with exams plus nops. That was an unusual experience, a funny battle towards final success :) Some notes: 1. Hardcoded nops template. It is a big problem that nops template is hardcoded in `make_nops_template` function. For cyrillic letters we need `\usepackage[russian]{babel}` or polyglossia equivalent. But it is not possible to include some arbitrary latex code into template. Our solution: we modified the internals of `make_nops_template` function. Basically we inserted `\input{add_in.tex}` there and worked with this `add_in.tex` and patched function. 2. Real life is not smooth. Two functions `nops_scan` and `nops_eval` suppose that everything is going smooth. That is a WRONG assumption! Only in ideal world we will have just two lines of code: `nops_scan` and `nops_eval`. In real life we need manual correction between these two lines. So the workflow may be: A. Scan B. Possible manual interaction B1. Check validity B2. Correct validity errors manually C. Evaluate automatically No help is provided for the step B. We need helper functions for the step B. 3. Bad recognition of exam id, scrambling Maybe it is possible to put QR code there? It is better suited for automatic recognition. Our solution was to put different variants into separate folders and override exam id recognition completely! We have just inferred the exam id based on the folder name :) How can we help user to interact with results manually? 1. Output of scan stage should be a data frame. User can correct manually misrecognized student ID or exam_id using some spreadsheet software. It is easy to export a data frame to csv, correct it and import back. But now the scan stage output is a character vector, which is not so easy to edit. We wrote our function to transform current output into data frame with columns: filename, student_id, exam_id, scrambling, q_no, given_answer. 2. Output of scan should be complete even in the case of error. If an answer sheet is not recognized scan stage should output a valid data frame! Now we have a word "ERROR" in character string. That's bad. We should have 30 rows (one for each question) added in our standard data frame. Like bad_file.png, NA, NA, 1, NA bad_file.png, NA, NA, 2, NA bad_file.png, NA, NA, 3, NA ... bad_file.png, NA, NA, 30, NA Why 30 rows? If the answer page is completely spoiled then it would be easier to enter answers manually!!! 3. We should have helper functions to check validity and make a report. * Check whether there NA in data frame. * Check whether there are double students id. * Check whether there student ids not in official students list * Check whether exam id is from official exam ID list * Check whether there are multiple crosses recognized for single choice questions * Check whether the number of scanned exams equals to the number of png files * some more? The report may be a data frame with columns filename, student_id, exam_id, scrambling, problem_description bad_file.png, 34, 56, 30, "double students id" wrong_file.png, 34, 58, 30, "double students id" very_bad.png, NA, NA, NA, "not recognized at all" not_so_bad.png, 39, 111111, 30, "wrongly recognized exam id" happy.png, 27, 39, 30, "OK" may_be_wrong.png, 33, 34, 56, "multiple answers to single choice questions" ... 4. Correspondingly `nops_evaluate` should take data frame input. One may leave character vector input as compatibility option. 5. Provide an outline of these steps in documentation or even function call One may even include some other practical advises. For example: Our scanners produce jpegs. We convert them into png using imagemagick. But the default installation of imagemagick on ubuntu has extremely low memory limit specified. So we have commented out memory limits in `policy.xml` imagemagick setup. Russian dcf file is here https://github.com/bdemeshev/exams/tree/master/inst/nops Thanks for a great package! from Russia with love :) |