Beyond Proficiency

Annex A. The PIAAC LogDataAnalyzer

Log files for 18 countries that participated in the first round of the Survey of Adult Skills, a product of the Programme for the International Assessment of Adult Competencies (PIAAC) (hereafter referred to as “PIAAC”) in 2011/12 have recently been released and can be downloaded from the German Social Science Infrastructure Services (GESIS) Data Catalogue (OECD, 2017[1]).1 The files have been fully anonymised to prevent identification of individual respondents. The records can nevertheless be matched with information already available in the PIAAC Public Use File, which contains the individual answers to the background questionnaire, as well as the performance of test-takers in the PIAAC assessment.

To facilitate the analysis of log files, the OECD has made available the LogDataAnalyzer (LDA), a software programme developed by GESIS. The LDA can be used to: 1) extract a number of predefined variables from the log files (which are in xml format); 2) export these variables in an external file, in txt format, which can be easily imported in the majority of software programmes used for statistical analysis; 3) compute and export descriptive statistics of the predefined variables; and 4) graphically visualise the predefined variables.

In particular, users can generate and extract the following variables:

• number of using cancel button

• number of using help menu

• time on task

• time till the first interaction

• final response

• number of switching environment

• sequence of switching environment

• number of highlight events

• time since last answer interaction

• number of created e-mails

• sequence of viewed e-mails

• number of different e-mail views

• number of revisited e-mails

• number of e-mail views

• sequence of visited web pages

• time-sequence of time spent on web pages

• number of different page visits

• number of page visits

• number of page revisits.

Figure A A.1. Initial screen of the LDA

Once the files are loaded, it is possible to select both the items (Figure A A.2) and the variables (Figure A A.3) to be analysed. At this stage, it will become apparent that not all variables are available for all items, so available variables will depend on the items selected in the first step. The selected variables, once extracted, can be exported in the form of a text file that can then be imported easily into other software.

Figure A A.2. Selection of items in the LDA

Figure A A.3. Selection of variables in the LDA

Finally, it is possible to compute simple descriptive statistics of the variables of interest, and display them either numerically or graphically (Figure A A.4).

Figure A A.4. Analysis of variables in the LDA

Full documentation of the content of the log files is available, including information on the content of the items, which is essential to properly interpret the variables extracted. Full access is restricted, as the documentation concerns items that have not yet been used in the assessment and should therefore be treated confidentially. Interested researchers should submit an application form, including a signed confidentiality agreement, to the OECD contact officer at [e-mail].

Reference

[1] OECD (2017), Programme for the International Assessment of Adult Competencies (PIAAC), Log Files, GESIS Data Archive, Cologne, http://dx.doi.org/10.4232/1.12955.

Note

← 1.  More information and documentation is available at www.oecd.org/skills/piaac/log-file/.