Relevance for Complex Systems Knowledge
The production of the model went through a trial-and-error process during the first 2 years of the project. The initial iteration involved a hierarchical approach based on Bloom’s taxonomy. Each of the 9 competencies:
- entrepreneurship and innovation,
- sustainability and social commitment,
- foreign language skills,
- effective oral and written communication,
- teamwork,
- information literacy,
- autonomous learning,
- appropriate attitude towards work,
- and reasoning,
was defined in terms of three-level learning outcomes, based on the “knowledge”, “comprehension”, and “application” domain levels in Bloom’s model. Level 1 of a competency was developed in first-year courses, level 2 in the second-year courses, and level 3 in third- and fourth-year courses. A coordinator was also assigned to each competency to help course designers and lecturers in designing learning activities to practice and assessing the results. Each subject had to provide two marks: the overall grade of the subject and the grade corresponding to the professional competency. This approach was difficult to carry out as domain levels are too general to define a competency.
The revised model uses competency maps to define learning outcomes instead. A competency map first defines a competency in terms of competency units (aspects or dimensions of the competency), which in turn determine the expected learning outcomes for each domain level. This approach resulted in a competency map in which specific learning outcomes were clearly defined for the corresponding competency units and domain levels and could be properly assigned at a suitable subject. All the professional competences are combined in a global competency map, which displays the three-domain level for each competency unit and professional competency. Each competency coordinator was in charge of defining learning outcomes and competency units and assigning specific learning outcomes to subjects. Then, authors started a procedure to integrate all competencies in the degree subjects through meetings with course coordinators.
The second research question was addressed using the students’ assessment results for each professional competency, the final grade of a subject is rated between 0 and 10 whereas ach professional competency gets a grade from A to D. The result is that more than 80% of evaluated students reached high scores (A or B), but the paper can’t provide a measure of “improvement” using the methodology because there’s no control group.
Research question 3 was also addressed in a qualitative way providing a questionnaire for students’ opinion on professional competencies and course satisfaction. More than 83% of students clearly feel that professional competencies are important for the development of their professional activity, also this perception strengthened over the project timespan.
Research question 4 is addressed through a survey provided to 2000 employers from Spanish companies on the employability of graduates from 2011 to 2015. The qualitative results show that the most valued school is the Barcelona School of Informatics, but the paper don’t provide more information on how this result is achieved (moreover the graduates from 2011 to 2012 wouldn’t be a part of the project, making the answer to this research question a bit weak).
Research question 5 is the only one addressed using a quantitative analysis method. The final academic grade was displayed vs each final professional competency grade (this time rated from 1 to 4) using density plots to observe potential levels of association. The grades of the nine professional competencies presented one of three patterns: either a triangular distribution, a logarithmic distribution, or a linear distribution. This means that the students with a high academic final grade never get low grades in any of the professional competencies.
None of the competency grades presented a normal distribution (they didn’t pass the Shapiro-Wilk test), thus it was not possible to calculate the Pearson correlation coefficient (linear relationship between two continuous variables) between the academic final grade and the professional competency grade. Only the average grade of all professional competencies followed a normal distribution (albeit with a p value very close to the threshold). The level of association is instead calculated using the Spearman correlation (which evaluates only the monotonic relationship) with low results for most of the professional competencies. The general conclusions that can be drawn are that a high final academic grade implies high competency grade, whereas the opposite is not necessarily true. Only the average grade for all the professional competencies shows a high level of association with the final academic grade, meaning that final academic grade implies high average competencies grade, and vice versa (the statement is still probably too farfetched considering that the p value is very close to being not significant).
