Academia.eduAcademia.edu
European Journal of Education Studies ISSN: 2501 - 1111 ISSN-L: 2501 - 1111 Available on-line at: www.oapub.org/edu Volume 3 │ Issue 1 │ 2017 doi: 10.5281/zenodo.223834 SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH Mile Ilić1, Ljiljana Jerković2 Ph.D, Full professor of specialised scientific disciplines of Didactics, 1 General Pedagogy and Class Teaching Methods Faculty of Philosophy Banja Luka, University of Banja Luka Bosnia and Herzegovina MA, Teaching assistant of specialised scientific disciplines of Didactics and General Pedagogy 2 Faculty of Philosophy Banja Luka, University of Banja Luka Bosnia and Herzegovina Abstract: The experimental method is more exact compared to other quantitative methods, although it can have drawbacks, related to the positivist epistemological position. Determining exactly the educational effects of existing, innovative and new pedagogical concepts, programmes, systems, models, methods and instruments commands the use of experiments in pedagogical research. If completed pedagogical research projects are analysed, the conclusion is experiments are used much less frequently than other methods. This study determines the prevalence of parallel-group designs as compared to how frequently other experimental designs are used. A representative sample of scientific and professional papers was analysed and it was ascertained that the conducted experiments partly satisfy relevant theoretical and methodological criteria. It is evident that result reliability when using the experimental method is still relatively low, which may have negative effects on the development of pedagogical sciences and related scientific disciplines, as well as on scientifically grounded innovation of the teaching and learning process and enhancement of the educational process. Hence, it is crucial to use multi-method research approaches (employing the experimental method as appropriate, depending on the research problem) in preparing (and approving) doctoral dissertations, writing reviews and publishing research papers. Copyright © The Author(s). All Rights Reserved. © 2015 – 2017 Open Access Publishing Group 112 Mile Ilić, Ljiljana Jerković SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH Keywords: experiment, experimental design, parallel-group experimental design, pedagogical research, multi-method research approach Introduction The experimental method is indisputably of great value for pedagogical research. The goal of experiments is not only to establish the cause and effect relationships between the studied phenomena, but also to study the effects of new concepts, procedures, strategies, methods, models and systems of work on teaching and learning and the educational process. The experimental method, as used in pedagogical research, is not an end in itself. Its purpose is to increase and expand the scope of knowledge of pedagogy, thereby enhancing the teaching and learning process and educational activity. Similar to many other research methods, the experimental method has its advantages and limitations. The value of an experiment depends on the extent the researcher is capable of and can expend the effort required to ensure the right experimental conditions, select the most appropriate design, successfully carry out the right experimental procedure, and is also ready to surmount difficulties accompanying experimentation in this field, arising from the complex nature of pedagogical sciences, as well as their other characteristics (Kocić, , p. . Therefore, the experiment is a highly sophisticated method whose effectiveness will depend on the researcher’s readiness to take account of all its advantages and potential weaknesses in the process of application, while studying a specific problem. Some potential benefits of using the experimental method in pedagogical research are: - The researcher does not need to wait for the right opportunity to conduct research, but can instead create such opportunities; - Research results can be verified, i.e., an experiment can be repeated; - In comparison with other methods, it allows the establishment of the cause and effect relationships between the independent variable and the dependent variable with greater certainty, as long as it is applied with a degree of caution; - It offers the possibility of stricter and more efficient control of the research conditions (ibid., pp. 136-137). Quantitative research methods, which also include the experimental method, are grounded in the philosophy of logical empiricism or positivism. Essentially, logical empiricism postulates that certain kinds of reliable conclusions can come from experience alone (Cohen, Manion & Morrison, 2007, p. 10). Accordingly, only experiential facts can be the subject of scientific cognizance. ”ogdan Бe:ić claims that, according to the philosophy of logical positivism, the main subject of all empirical sciences European Journal of Education Studies - Volume 3 │ Issue 1 │ 2017 113 Mile Ilić, Ljiljana Jerković SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH is empirical facts, i.e., sensations, sensuous impressions and perceptions, their direct experience, that is, cognizance in the form of factual judgements Бe:ić, , p. . The main weakness of the experimental method (as well as of other quantitative methods) is attributed to the positivist epistemological position, which makes it impossible to fully understand the essence of the studied phenomenon, as it disregards the situational context. Slavo Kukić claims that a quantitative approach, which is also a characteristic of the experimental method, is always partial, because it tests only a certain number of hypotheses, and not the whole phenomenon Kukić, , p. . Stanislav Fajgelj stresses that experiments cannot be successfully conducted Fajgelj, , p. 223). An experiment can never be fully controlled, as planned by the design. It is difficult to preclude interaction between group members. Also, junior researchers often proceed at will (ibid., p. 223). Pedagogical literature on the topic of methodology lists the following kinds of experiment designs: 1. One-group design; 2. Parallel-group design; 3. Factor rotation design/factor analysis; and 4. Ex post facto design (”anгur & Potkonjak, Mandić, Savićević, One-group designs are the kind of experiment designs in which one or more factors are introduced to one group, and its or their effects monitored relative to the original examination Mandić, . Parallel-group designs include working with at least two groups of research subjects. An experiment factor is introduced in working with one of the groups (e.g., a new system of teaching and learning, procedure, programme/syllabus), whereas the other group is the control group, which the researcher works with using the traditional (usual) approach ”anгur & Potkonjak, . In factor rotation designs, all experiment factors are successively introduced to all groups of research subjects ”anгur & Potkonjak, Mandić, . Ex post facto designs study the cause and effect relationships that exist between pedagogical phenomena or in the educational process itself, without introducing any experiment factors. The experiment factor already exists, as it was previously introduced by someone else ”anгur & Potkonjak, . Practically, all of the designs listed above have their virtues and downsides. According to pedagogical methodology literature, parallel-group designs are conducted the most frequently and are also the most reliable. One of the aims of this study was to determine the frequency of use of this type of experiment design. European Journal of Education Studies - Volume 3 │ Issue 1 │ 2017 114 Mile Ilić, Ljiljana Jerković SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH Methodology Research goal The goal of this research is to ascertain the prevalence and exactness of the experimental method as used in pedagogical research. Hypotheses 1. It is expected that experiments are conducted significantly less often in pedagogical research than the descriptive method is employed; 2. It is assumed that the frequency of use of parallel-group designs in pedagogical research is more statistically significant compared to other experimental designs; 3. For most of the pedagogical research studied herein, the conducted experiments partly satisfy the theoretical and methodological criteria listed below: 3.1. Thematic relevance of the experiment; 3.2. Theoretical soundness of the experiment; 3.3. Extent of reliance on previous research; 3.4. Explicitness of the experiment factor (independent variable); 3.5. Concreteness of the dependent variables; 3.6. Equitability of relevant variables; 3.7. Relevance of research tools relative to the research theme, content and problem; 3.8. Explicit description of the experimenter training programme; 3.9. Duration of the experiment; 3.10. Appropriateness of the statistical procedures used for data processing; 3.11. Full interpretation of the experiment results; 3.12. Degree of verification of the suggested hypothesis; 3.13. Consistency in drawing conclusions (making scientific generalizations); 3.14. Degree of contribution to science (novelty); 3.15. Explicit description of result applicability; and 3.16. Presentation of identified problems to be dealt with in future research. Research tool The research tool used for the needs of this study was the protocol of analysis of the experimental method appropriateness for pedagogical research relative to content and the research problem. Study sample The study sample included 147 scientific works (31 doctoral dissertations, 66 master theses, 19 research monographs and 31 research papers) and 63 professional (master) papers, of which 105 were the result of empirical research. European Journal of Education Studies - Volume 3 │ Issue 1 │ 2017 115 Mile Ilić, Ljiljana Jerković SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH The sample was taken from the host of research works published in the framework of activity of two teacher training faculties – The Faculty of Philosophy in Banja Luka and The Faculty of Philosophy in Pale. The sample comprised approximately 80% of the doctoral dissertations and master theses (both pre-Bologna and Bologna systems) from the period between 2000 and 2015, as well as over 10% of the research monographs and papers written in the same period by professors and instructors from the two institutions. The sample included approximately 80% of all works from the period 2000 to 2015. The subsample comprised 30 research works in which the experimental method was used. The experiment was the unit of analysis of the work contents. Results and discussion While acknowledging the importance of the experiment as a research method, this study attempts to ascertain its frequency compared to the descriptive method as used in pedagogical research. Table 1: Experimental method in pedagogical research Method N % Experimental 30 28.57 Descriptive 75 71.43 Total 105 100 χ2 p 19.29 0.00 According to the indicators presented in Table 1 and the chi-squared distribution χ = . p= . , it is possible to conclude that in terms of statistical significance, the experimental method is far less frequently used in didactic and methodological research (28.57%) than the descriptive method (71.43%), which confirms our first auxiliary hypothesis. The fact that the descriptive method prevails in pedagogical research suggests that the situation in the classroom and educational reality is mostly described as it is. Insights like this can be stimulating for further research, including experimental research; sometimes, they can be developmental and strategic in nature. The purpose of the descriptive method is not simply to describe a phenomenon under consideration, but to interpret, compare and contrast data, with the aim of making scientific generalisations. Such generalisations are often declarative. Does it suffice to simply describe the situation as it is, which is also quite often unfavourable, make declarative generalisations and identify new issues, and by doing so make improvements in the area of teaching and learning and educational activity? Can description be used to effect scientifically grounded change in teaching and learning and the educational process? We were interested in the extent to which specific experiment designs are used in pedagogical research. European Journal of Education Studies - Volume 3 │ Issue 1 │ 2017 116 Mile Ilić, Ljiljana Jerković SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH Table 2: Prevalence of different experiment designs in pedagogical research Experiment designs N % Parallel-group design 28 93.33 2 6.67 30 100 Other experiment designs Total χ2 p 22.53 0.00 According to the statistical indicators shown in Table 2, the prevalence of parallel-group designs (93.33%) is evident when compared to other designs, corroborating our second auxiliary hypothesis. Parallel-group designs are the most reliable, since the use of the control group allows the researcher to determine the impact of the experiment factor (a new system of teaching and learning, a new procedure, strategy, etc.), on condition the experimental group and the control group are equivalent. As it is impossible to achieve absolute group equivalence, parallel-group designs cannot be considered fully reliable. We were interested in seeing to what extent research projects meet the relevant theoretical and methodological criteria. Table 3: Scientific and methodological soundness of the experimental method in pedagogical research Criteria of scientific and methodological soundness of experiments Identified degree of fulfilment of criteria does not satisfy the criteria barely refers to the criteria satisfy the criteria fulfil the criteria χ2 p N % N % N % N % 1. Theoretical soundness of the experiment 0 0 2 6.7 14 46,7 14 46.6 9.73 0.05 2. Thematic relevance of the experiment 0 0 5 16.7 0 0 25 83.3 7.80 0.05 3. Extent of reliance on previous research 3 10.0 7 23.3 6 20.0 14 46.7 5.00 0.29 4. Explicitness of the experiment factor (independent variable) 2 6.7 8 26.7 7 23.3 13 13.3 5.67 0.22 5. Concreteness of the dependent variables 1 3.3 4 13.3 8 26.7 17 56.7 7.67 0.10 6. Equitability of relevant variables 2 6.7 3 10.0 5 16.7 20 66.6 10.00 0.05 7. Relevance of research tools relative to the research theme, content and problem 0 0 0 0 7 23.3 23 76.7 5.40 0.07 22 73.3 5 16.7 2 6.7 1 3.3 38.53 0,00 4 13.3 10 33.3 11 36.67 5 16.6 4.93 0.18 0 0 1 3.3 8 26.7 21 70.0 7.60 0.05 8. Explicit description of the experimenter training programme 9. Appropriateness duration of the experiment to the complexity of the research problem 10. Appropriateness of the statistical procedures used for data processing European Journal of Education Studies - Volume 3 │ Issue 1 │ 2017 117 Mile Ilić, Ljiljana Jerković SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH 11. Full interpretation of the experiment results 0 0 4 13.4 13 43.3 13 43.3 8.13 0.05 12. Degree of verification of the suggested hypothesis 0 0 1 3.3 10 33.3 19 63.4 10.80 0.05 0 0 7 23.3 9 30.0 14 46.7 1.46 0.69 1 3.3 9 30.0 12 40.0 8 26.7 13.00 0.01 11 36.7 7 23.3 3 10.0 9 30.0 6.66 0.15 20 66.7 1 3.3 3 10.0 6 20.0 42.66 0.00 13. Consistency in drawing conclusions (making scientific generalisations) 14. Degree of contribution to science (novelty) 15. Explicit description of result applicability 16. Presentation of identified problems to be dealt with in future research According to the data given in Table 3, the majority of the papers analysed satisfy or fulfil the following requirements of theoretical and methodological soundness of the experiment: theoretical soundness and thematic relevance of the experiment; equitability of the relevant variables; relevance of research tools relative to the research content and problems; appropriateness of the statistical procedures used; comprehensive interpretation of the research results; verification of the proposed hypotheses; and a degree of scientific novelty. The thematic relevance and theoretical soundness of the experiment, along with equitability of the relevant variables, reduce the possibility of improvisation and arbitrariness in education research. The results of this study show that the research projects investigated herein mainly meet the criteria stated in this paragraph. According to the indicators in the same table, the majority of the research tools as employed in the examined works satisfy or fulfil the criteria of relevance relative to the research content and problems, which makes this a barely detectable statistically significant difference. Adequate research tools are an important condition for the validity of research results. According to the data stated in Table 3, a significantly high number of the works meet to a certain degree the criterion of appropriateness of statistical procedures relative to the studied phenomenon χ = . p= . . The research results may also be considered comprehensively interpreted χ = . p= . . Hypothesis verification allows the researcher to set a framework within which to make conclusions and generalisations. A great majority of the works examined state whether the set hypothesis has been adopted or rejected, compounded with relevant fact-based explanations χ = . p= . . According to the statistical indicators presented in the same table, for the majority of the works examined there is no significant difference in terms of the following criteria of theoretical and methodological soundness of the experiment: European Journal of Education Studies - Volume 3 │ Issue 1 │ 2017 118 Mile Ilić, Ljiljana Jerković SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH reliance on previously conducted research; explicit description of the experiment factor; description of the dependent variables; an appropriate duration of the experiment; complexity of the research problem; consistency in drawing scientific generalisations; and stating research result applicability. The statistical indicators show that every third experiment does not rely on the results of previously conducted empirical research, which means they do not sufficiently meet the requirement of continuity meant to increase and expand scientific knowledge. Failure to take into consideration the results of previously conducted research often leads to researching previously researched problems and discovering previously discovered truths . One-third of the works is merely close to or does not satisfy the criterion of explicitness of the experiment factor. Two-thirds of them satisfy or fulfil this criterion. Failure on the part of the researcher to explicate the independent variable is conducive to doubt as to the exactness of the research and precludes the experiment from being repeated by other researchers. The dependent variable has to satisfy two important requirements. It is important that it be an adequate representation of the research construct and reliably measurable and perceivable (Milas, 2009, p. 113). Every sixth study or work does not satisfy or barely refers to the criterion of precise definition of dependent variables. The experiment duration depends on the complexity of the experiment factor. The more complex and sophisticated the factor, the longer the experiment. Studying the impact of a factor on rote learning is certainly different from examining its influence on student-held values. The data presented in the table indicate that for approximately one half of the pedagogical works examined in this study the experiment duration corresponded to the phenomenon complexity, whereas for the other half it did not. Also, the majority of these pedagogical works satisfy or fulfil the criterion of consistency in drawing scientific generalisations. The scientific generalisations made crown the conducted research. As far as contemporary didactic and methodological research is concerned, it is absolutely necessary to refer to the possible applications of research results in the classroom and educational practice, as well as their dissemination. More than a half of the studies or works analysed (approximately 60%) do not write about or barely mention the applicability of their research results. The ultimate goal of research is not scientific cognizance for the sake of scientific cognizance , but rather the enhancement of teaching, learning and educational activity. Pedagogical research which does not address the issues of applicability and dissemination of its results is of questionable value. According to the data in Table 3, the majority of the analysed papers and works do not satisfy the following criteria of theoretical and methodological soundness of the experimental method: explicitness of the experiment factor (independent variable) and raising new issues and problems. Over 85% of the papers and works do not provide a European Journal of Education Studies - Volume 3 │ Issue 1 │ 2017 119 Mile Ilić, Ljiljana Jerković SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH description of a specific experimenter training programme. These data may signal that an experiment may have been arbitrarily conducted, which undoubtedly affected the validity of the obtained results. Apart from training the experimenters to carry out the experiment, it is worthwhile to hold weekly meetings of the researcher and the experimenters, where problems can be dealt with jointly and notes taken with regard to how frequently specific problems appear. This type of information can be useful when evaluating the experiment results, which may or may not tally with the researcher’s expectations (McGowan, 2011, p. 5). There is a statistically significant difference between the analysed works in terms of presenting problems identified to be addressed in further research. Namely, the number of pedagogical experimental research projects which do not write about problems to be addressed by researchers in the future (over 70%) is greater than those that do. Such a treatment of research results on the part of researchers does not contribute to the advancement of scientific knowledge. In accordance with the above analysis, it may be concluded that the third hypothesis has been confirmed, i.e., the conducted experiments partly satisfy the necessary theoretical and methodological criteria. Conclusions Researchers’ preferring other methods to the experimental method as used in pedagogy undoubtedly has negative consequences on the development of the system of scientific pedagogical knowledge, the innovation of teaching and learning and the enhancement of the education process. While it is necessary to bear in mind the potential limitations of the experimental method, its advantages compared to other research methods must not be disregarded either. In light of the results of this study into the frequency and quality of the experimental method as employed in pedagogical research, it may be considered necessary to do the following: 1. Significantly increase the number of thematically, theoretically and methodologically appropriate experiments that will optimally take account of previously conducted empirical research, clearly describe the experiment factor (independent variable), present, in detail, the procedure and content of experimenter and researcher training, make explicit the areas of application of the research results and state the remaining problems to be addressed by research in the future. 2. Significantly improve the methodology of experimental research, thereby optimising its contribution to the advancement of pedagogical scientific disciplines, which will have an effect in the sense of enhancement, grounded in European Journal of Education Studies - Volume 3 │ Issue 1 │ 2017 120 Mile Ilić, Ljiljana Jerković SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH science, of the process and outcomes of teaching and learning activity and the education process. 3. Interpret research results employing both quantitative and qualitative approaches, by making more permanent and higher quality scientific syntheses; 4. Take into consideration the above presented methodological criteria of valuation of the exactness of experimental research, which may help lessen the use of pseudo-methods and the gaining of quasi-scientific results and insights in experimental pedagogical research. References а ђ , . и к њак, . . М л гија аг гиј . а а з а шких ш а аЈ а иј . 2. Cohen, L., Manion, L. & Morrison, K. (2007). Metode u istraživanja u obrazovanjuc (prevod petog izdanja). Zagreb: Naklada Slap. 3. Fajgelj, S. (2014). Metode istraživanja ponašanja (V dopunjeno izdanje). Beograd: Centar za primenjenu psihologiju 4. Kocić, Lj. . Pedagoški eksperiment – karakteristike i mogućnosti. Beograd: Institut za pedago:ka istraživanja i Prosveta. 5. Kukić, S. . Metodologija znanstvenog istraživanja. Sarajevo: Sarajevo publishing. 6. а ић, . .М л гија на чн г а а. ања ка Ака ија а ка и ј и ик к . 7. McGowan, H. M. (2011). Planning a Comparative Experiment in Educational Settings. Journal of Statistics Education, Volume 19, Number 2. 8. Milas, G. (2009). Istraživačke metode u psihologiji i drugim društvenim znanostima. Zagreb „Naklada Slap . 9. Mužić, V. . Metodologija pedagoškog istraživanja. Sarajevo: Zavod za udžbenike i nastavna sredstva. 10. Savićević, D. . Metodologija istraživanja u vaspitanju i obrazovanju. Vranje: Univerzitet u Ni:u Učiteljski fakultet u Vranju. 11. Бe:ić, ”. . Osnovi metodologije društvenih nauka. ”eograd Naučna knjiga. 1. European Journal of Education Studies - Volume 3 │ Issue 1 │ 2017 121 Mile Ilić, Ljiljana Jerković SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH Creative Commons licensing terms Author(s) will retain the copyright of their published articles agreeing that a Creative Commons Attribution 4.0 International License (CC BY 4.0) terms will be applied to their work. Under the terms of this license, no permission is required from the author(s) or publisher for members of the community to copy, distribute, transmit or adapt the article content, providing a proper, prominent and unambiguous attribution to the authors in a manner that makes clear that the materials are being reused under permission of a Creative Commons License. Views, opinions and conclusions expressed in this research article are views, opinions and conclusions of the author(s). Open Access Publishing Group and European Journal of Education Studies shall not be responsible or answerable for any loss, damage or liability caused in relation to/arising out of conflicts of interest, copyright violations and inappropriate or inaccurate use of any kind content related or integrated into the research work. All the published works are meeting the Open Access Publishing requirements and can be freely accessed, shared, modified, distributed and used in educational, commercial and non-commercial purposes under a Creative Commons Attribution 4.0 International License (CC BY 4.0). European Journal of Education Studies - Volume 3 │ Issue 1 │ 2017 122