European Journal of Education Studies
ISSN: 2501 - 1111
ISSN-L: 2501 - 1111
Available on-line at: www.oapub.org/edu
Volume 3 │ Issue 1 │ 2017
doi: 10.5281/zenodo.223834
SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF
EXPERIMENTS IN PEDAGOGICAL RESEARCH
Mile Ilić1, Ljiljana Jerković2
Ph.D, Full professor of specialised scientific disciplines of Didactics,
1
General Pedagogy and Class Teaching Methods
Faculty of Philosophy Banja Luka, University of Banja Luka
Bosnia and Herzegovina
MA, Teaching assistant of specialised scientific disciplines of Didactics and General Pedagogy
2
Faculty of Philosophy Banja Luka, University of Banja Luka
Bosnia and Herzegovina
Abstract:
The experimental method is more exact compared to other quantitative methods,
although it can have drawbacks, related to the positivist epistemological position.
Determining exactly the educational effects of existing, innovative and new pedagogical
concepts, programmes, systems, models, methods and instruments commands the use
of experiments in pedagogical research. If completed pedagogical research projects are
analysed, the conclusion is experiments are used much less frequently than other
methods.
This study determines the prevalence of parallel-group designs as compared to
how frequently other experimental designs are used. A representative sample of
scientific and professional papers was analysed and it was ascertained that the
conducted experiments partly satisfy relevant theoretical and methodological criteria. It
is evident that result reliability when using the experimental method is still relatively
low, which may have negative effects on the development of pedagogical sciences and
related scientific disciplines, as well as on scientifically grounded innovation of the
teaching and learning process and enhancement of the educational process. Hence, it is
crucial to use multi-method research approaches (employing the experimental method
as appropriate, depending on the research problem) in preparing (and approving)
doctoral dissertations, writing reviews and publishing research papers.
Copyright © The Author(s). All Rights Reserved.
© 2015 – 2017 Open Access Publishing Group
112
Mile Ilić, Ljiljana Jerković
SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH
Keywords: experiment, experimental design, parallel-group experimental design,
pedagogical research, multi-method research approach
Introduction
The experimental method is indisputably of great value for pedagogical research. The
goal of experiments is not only to establish the cause and effect relationships between
the studied phenomena, but also to study the effects of new concepts, procedures,
strategies, methods, models and systems of work on teaching and learning and the
educational process. The experimental method, as used in pedagogical research, is not
an end in itself. Its purpose is to increase and expand the scope of knowledge of
pedagogy, thereby enhancing the teaching and learning process and educational
activity.
Similar to many other research methods, the experimental method has its
advantages and limitations. The value of an experiment depends on the extent the
researcher is capable of and can expend the effort required to ensure the right experimental
conditions, select the most appropriate design, successfully carry out the right experimental
procedure, and is also ready to surmount difficulties accompanying experimentation in this field,
arising from the complex nature of pedagogical sciences, as well as their other characteristics
(Kocić,
, p.
. Therefore, the experiment is a highly sophisticated method whose
effectiveness will depend on the researcher’s readiness to take account of all its
advantages and potential weaknesses in the process of application, while studying a
specific problem.
Some potential benefits of using the experimental method in pedagogical
research are:
- The researcher does not need to wait for the right opportunity to conduct
research, but can instead create such opportunities;
- Research results can be verified, i.e., an experiment can be repeated;
- In comparison with other methods, it allows the establishment of the cause and
effect relationships between the independent variable and the dependent
variable with greater certainty, as long as it is applied with a degree of caution;
- It offers the possibility of stricter and more efficient control of the research
conditions (ibid., pp. 136-137).
Quantitative research methods, which also include the experimental method, are
grounded in the philosophy of logical empiricism or positivism. Essentially, logical
empiricism postulates that certain kinds of reliable conclusions can come from
experience alone (Cohen, Manion & Morrison, 2007, p. 10). Accordingly, only
experiential facts can be the subject of scientific cognizance. ”ogdan Бe:ić claims that,
according to the philosophy of logical positivism, the main subject of all empirical sciences
European Journal of Education Studies - Volume 3 │ Issue 1 │ 2017
113
Mile Ilić, Ljiljana Jerković
SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH
is empirical facts, i.e., sensations, sensuous impressions and perceptions, their direct experience,
that is, cognizance in the form of factual judgements Бe:ić,
, p.
.
The main weakness of the experimental method (as well as of other quantitative
methods) is attributed to the positivist epistemological position, which makes it
impossible to fully understand the essence of the studied phenomenon, as it disregards
the situational context. Slavo Kukić claims that a quantitative approach, which is also a
characteristic of the experimental method, is always partial, because it tests only a
certain number of hypotheses, and not the whole phenomenon Kukić,
, p.
.
Stanislav Fajgelj stresses that experiments cannot be successfully conducted Fajgelj,
,
p. 223). An experiment can never be fully controlled, as planned by the design. It is
difficult to preclude interaction between group members. Also, junior researchers often
proceed at will (ibid., p. 223).
Pedagogical literature on the topic of methodology lists the following kinds of
experiment designs:
1. One-group design;
2. Parallel-group design;
3. Factor rotation design/factor analysis; and
4. Ex post facto design (”anгur & Potkonjak,
Mandić,
Savićević,
One-group designs are the kind of experiment designs in which one or more
factors are introduced to one group, and its or their effects monitored relative to the
original examination Mandić,
.
Parallel-group designs include working with at least two groups of research
subjects. An experiment factor is introduced in working with one of the groups (e.g., a
new system of teaching and learning, procedure, programme/syllabus), whereas the
other group is the control group, which the researcher works with using the traditional
(usual) approach ”anгur & Potkonjak,
.
In factor rotation designs, all experiment factors are successively introduced to
all groups of research subjects ”anгur & Potkonjak,
Mandić,
.
Ex post facto designs study the cause and effect relationships that exist between
pedagogical phenomena or in the educational process itself, without introducing any
experiment factors. The experiment factor already exists, as it was previously
introduced by someone else ”anгur & Potkonjak,
.
Practically, all of the designs listed above have their virtues and downsides.
According to pedagogical methodology literature, parallel-group designs are conducted
the most frequently and are also the most reliable. One of the aims of this study was to
determine the frequency of use of this type of experiment design.
European Journal of Education Studies - Volume 3 │ Issue 1 │ 2017
114
Mile Ilić, Ljiljana Jerković
SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH
Methodology
Research goal
The goal of this research is to ascertain the prevalence and exactness of the experimental
method as used in pedagogical research.
Hypotheses
1. It is expected that experiments are conducted significantly less often in
pedagogical research than the descriptive method is employed;
2. It is assumed that the frequency of use of parallel-group designs in pedagogical
research is more statistically significant compared to other experimental designs;
3. For most of the pedagogical research studied herein, the conducted experiments
partly satisfy the theoretical and methodological criteria listed below:
3.1. Thematic relevance of the experiment;
3.2. Theoretical soundness of the experiment;
3.3. Extent of reliance on previous research;
3.4. Explicitness of the experiment factor (independent variable);
3.5. Concreteness of the dependent variables;
3.6. Equitability of relevant variables;
3.7. Relevance of research tools relative to the research theme, content and
problem;
3.8. Explicit description of the experimenter training programme;
3.9. Duration of the experiment;
3.10. Appropriateness of the statistical procedures used for data processing;
3.11. Full interpretation of the experiment results;
3.12. Degree of verification of the suggested hypothesis;
3.13. Consistency in drawing conclusions (making scientific generalizations);
3.14. Degree of contribution to science (novelty);
3.15. Explicit description of result applicability; and
3.16. Presentation of identified problems to be dealt with in future research.
Research tool
The research tool used for the needs of this study was the protocol of analysis of the
experimental method appropriateness for pedagogical research relative to content and
the research problem.
Study sample
The study sample included 147 scientific works (31 doctoral dissertations, 66 master
theses, 19 research monographs and 31 research papers) and 63 professional (master)
papers, of which 105 were the result of empirical research.
European Journal of Education Studies - Volume 3 │ Issue 1 │ 2017
115
Mile Ilić, Ljiljana Jerković
SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH
The sample was taken from the host of research works published in the
framework of activity of two teacher training faculties – The Faculty of Philosophy in
Banja Luka and The Faculty of Philosophy in Pale. The sample comprised
approximately 80% of the doctoral dissertations and master theses (both pre-Bologna
and Bologna systems) from the period between 2000 and 2015, as well as over 10% of
the research monographs and papers written in the same period by professors and
instructors from the two institutions. The sample included approximately 80% of all
works from the period 2000 to 2015. The subsample comprised 30 research works in
which the experimental method was used. The experiment was the unit of analysis of
the work contents.
Results and discussion
While acknowledging the importance of the experiment as a research method, this
study attempts to ascertain its frequency compared to the descriptive method as used in
pedagogical research.
Table 1: Experimental method in pedagogical research
Method
N
%
Experimental
30
28.57
Descriptive
75
71.43
Total
105
100
χ2
p
19.29
0.00
According to the indicators presented in Table 1 and the chi-squared distribution
χ = .
p= . , it is possible to conclude that in terms of statistical significance, the
experimental method is far less frequently used in didactic and methodological research
(28.57%) than the descriptive method (71.43%), which confirms our first auxiliary
hypothesis. The fact that the descriptive method prevails in pedagogical research
suggests that the situation in the classroom and educational reality is mostly described
as it is. Insights like this can be stimulating for further research, including experimental
research; sometimes, they can be developmental and strategic in nature. The purpose of
the descriptive method is not simply to describe a phenomenon under consideration,
but to interpret, compare and contrast data, with the aim of making scientific
generalisations. Such generalisations are often declarative. Does it suffice to simply
describe the situation as it is, which is also quite often unfavourable, make declarative
generalisations and identify new issues, and by doing so make improvements in the
area of teaching and learning and educational activity? Can description be used to effect
scientifically grounded change in teaching and learning and the educational process?
We were interested in the extent to which specific experiment designs are used in
pedagogical research.
European Journal of Education Studies - Volume 3 │ Issue 1 │ 2017
116
Mile Ilić, Ljiljana Jerković
SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH
Table 2: Prevalence of different experiment designs in pedagogical research
Experiment designs
N
%
Parallel-group design
28
93.33
2
6.67
30
100
Other experiment designs
Total
χ2
p
22.53
0.00
According to the statistical indicators shown in Table 2, the prevalence of parallel-group
designs (93.33%) is evident when compared to other designs, corroborating our second
auxiliary hypothesis. Parallel-group designs are the most reliable, since the use of the
control group allows the researcher to determine the impact of the experiment factor (a
new system of teaching and learning, a new procedure, strategy, etc.), on condition the
experimental group and the control group are equivalent. As it is impossible to achieve
absolute group equivalence, parallel-group designs cannot be considered fully reliable.
We were interested in seeing to what extent research projects meet the relevant
theoretical and methodological criteria.
Table 3: Scientific and methodological soundness of the experimental method
in pedagogical research
Criteria of scientific and
methodological soundness of
experiments
Identified degree of fulfilment of criteria
does not
satisfy
the
criteria
barely refers
to
the criteria
satisfy
the
criteria
fulfil
the
criteria
χ2
p
N
%
N
%
N
%
N
%
1. Theoretical soundness of the
experiment
0
0
2
6.7
14
46,7
14
46.6
9.73
0.05
2. Thematic relevance of the
experiment
0
0
5
16.7
0
0
25
83.3
7.80
0.05
3. Extent of reliance on previous
research
3
10.0
7
23.3
6
20.0
14
46.7
5.00
0.29
4. Explicitness of the experiment
factor (independent variable)
2
6.7
8
26.7
7
23.3
13
13.3
5.67
0.22
5. Concreteness of the dependent
variables
1
3.3
4
13.3
8
26.7
17
56.7
7.67
0.10
6. Equitability of relevant variables
2
6.7
3
10.0
5
16.7
20
66.6
10.00
0.05
7. Relevance of research tools
relative to the research theme,
content and problem
0
0
0
0
7
23.3
23
76.7
5.40
0.07
22
73.3
5
16.7
2
6.7
1
3.3
38.53
0,00
4
13.3
10
33.3
11
36.67
5
16.6
4.93
0.18
0
0
1
3.3
8
26.7
21
70.0
7.60
0.05
8. Explicit description of the
experimenter training programme
9. Appropriateness
duration of the experiment to the
complexity of the research problem
10. Appropriateness of the statistical
procedures used for data processing
European Journal of Education Studies - Volume 3 │ Issue 1 │ 2017
117
Mile Ilić, Ljiljana Jerković
SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH
11. Full interpretation of the
experiment results
0
0
4
13.4
13
43.3
13
43.3
8.13
0.05
12. Degree of verification of the
suggested hypothesis
0
0
1
3.3
10
33.3
19
63.4
10.80
0.05
0
0
7
23.3
9
30.0
14
46.7
1.46
0.69
1
3.3
9
30.0
12
40.0
8
26.7
13.00
0.01
11
36.7
7
23.3
3
10.0
9
30.0
6.66
0.15
20
66.7
1
3.3
3
10.0
6
20.0
42.66
0.00
13. Consistency in drawing
conclusions (making scientific
generalisations)
14. Degree of contribution to science
(novelty)
15. Explicit description of result
applicability
16. Presentation of identified
problems to be dealt with in future
research
According to the data given in Table 3, the majority of the papers analysed satisfy or
fulfil the following requirements of theoretical and methodological soundness of the
experiment: theoretical soundness and thematic relevance of the experiment;
equitability of the relevant variables; relevance of research tools relative to the research
content and problems; appropriateness of the statistical procedures used;
comprehensive interpretation of the research results; verification of the proposed
hypotheses; and a degree of scientific novelty.
The thematic relevance and theoretical soundness of the experiment, along with
equitability of the relevant variables, reduce the possibility of improvisation and
arbitrariness in education research. The results of this study show that the research
projects investigated herein mainly meet the criteria stated in this paragraph.
According to the indicators in the same table, the majority of the research tools as
employed in the examined works satisfy or fulfil the criteria of relevance relative to the
research content and problems, which makes this a barely detectable statistically
significant difference. Adequate research tools are an important condition for the
validity of research results.
According to the data stated in Table 3, a significantly high number of the works
meet to a certain degree the criterion of appropriateness of statistical procedures
relative to the studied phenomenon χ = .
p= . . The research results may also be
considered comprehensively interpreted χ = .
p= . .
Hypothesis verification allows the researcher to set a framework within which to
make conclusions and generalisations. A great majority of the works examined state
whether the set hypothesis has been adopted or rejected, compounded with relevant
fact-based explanations χ = .
p= . .
According to the statistical indicators presented in the same table, for the
majority of the works examined there is no significant difference in terms of the
following criteria of theoretical and methodological soundness of the experiment:
European Journal of Education Studies - Volume 3 │ Issue 1 │ 2017
118
Mile Ilić, Ljiljana Jerković
SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH
reliance on previously conducted research; explicit description of the experiment factor;
description of the dependent variables; an appropriate duration of the experiment;
complexity of the research problem; consistency in drawing scientific generalisations;
and stating research result applicability.
The statistical indicators show that every third experiment does not rely on the
results of previously conducted empirical research, which means they do not
sufficiently meet the requirement of continuity meant to increase and expand scientific
knowledge. Failure to take into consideration the results of previously conducted
research often leads to researching previously researched problems and discovering
previously discovered truths .
One-third of the works is merely close to or does not satisfy the criterion of
explicitness of the experiment factor. Two-thirds of them satisfy or fulfil this criterion.
Failure on the part of the researcher to explicate the independent variable is conducive
to doubt as to the exactness of the research and precludes the experiment from being
repeated by other researchers.
The dependent variable has to satisfy two important requirements. It is
important that it be an adequate representation of the research construct and reliably
measurable and perceivable (Milas, 2009, p. 113). Every sixth study or work does not
satisfy or barely refers to the criterion of precise definition of dependent variables.
The experiment duration depends on the complexity of the experiment factor.
The more complex and sophisticated the factor, the longer the experiment. Studying the
impact of a factor on rote learning is certainly different from examining its influence on
student-held values. The data presented in the table indicate that for approximately one
half of the pedagogical works examined in this study the experiment duration
corresponded to the phenomenon complexity, whereas for the other half it did not.
Also, the majority of these pedagogical works satisfy or fulfil the criterion of
consistency in drawing scientific generalisations. The scientific generalisations made
crown the conducted research. As far as contemporary didactic and methodological
research is concerned, it is absolutely necessary to refer to the possible applications of
research results in the classroom and educational practice, as well as their
dissemination. More than a half of the studies or works analysed (approximately 60%)
do not write about or barely mention the applicability of their research results. The
ultimate goal of research is not scientific cognizance for the sake of scientific cognizance ,
but rather the enhancement of teaching, learning and educational activity. Pedagogical
research which does not address the issues of applicability and dissemination of its
results is of questionable value.
According to the data in Table 3, the majority of the analysed papers and works
do not satisfy the following criteria of theoretical and methodological soundness of the
experimental method: explicitness of the experiment factor (independent variable) and
raising new issues and problems. Over 85% of the papers and works do not provide a
European Journal of Education Studies - Volume 3 │ Issue 1 │ 2017
119
Mile Ilić, Ljiljana Jerković
SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH
description of a specific experimenter training programme. These data may signal that
an experiment may have been arbitrarily conducted, which undoubtedly affected the
validity of the obtained results. Apart from training the experimenters to carry out the
experiment, it is worthwhile to hold weekly meetings of the researcher and the
experimenters, where problems can be dealt with jointly and notes taken with regard to
how frequently specific problems appear. This type of information can be useful when
evaluating the experiment results, which may or may not tally with the researcher’s
expectations (McGowan, 2011, p. 5).
There is a statistically significant difference between the analysed works in terms
of presenting problems identified to be addressed in further research. Namely, the
number of pedagogical experimental research projects which do not write about
problems to be addressed by researchers in the future (over 70%) is greater than those
that do. Such a treatment of research results on the part of researchers does not
contribute to the advancement of scientific knowledge.
In accordance with the above analysis, it may be concluded that the third
hypothesis has been confirmed, i.e., the conducted experiments partly satisfy the
necessary theoretical and methodological criteria.
Conclusions
Researchers’ preferring other methods to the experimental method as used in pedagogy
undoubtedly has negative consequences on the development of the system of scientific
pedagogical knowledge, the innovation of teaching and learning and the enhancement
of the education process. While it is necessary to bear in mind the potential limitations
of the experimental method, its advantages compared to other research methods must
not be disregarded either.
In light of the results of this study into the frequency and quality of the
experimental method as employed in pedagogical research, it may be considered
necessary to do the following:
1. Significantly increase the number of thematically, theoretically and
methodologically appropriate experiments that will optimally take account of
previously conducted empirical research, clearly describe the experiment factor
(independent variable), present, in detail, the procedure and content of
experimenter and researcher training, make explicit the areas of application of
the research results and state the remaining problems to be addressed by
research in the future.
2. Significantly improve the methodology of experimental research, thereby
optimising its contribution to the advancement of pedagogical scientific
disciplines, which will have an effect in the sense of enhancement, grounded in
European Journal of Education Studies - Volume 3 │ Issue 1 │ 2017
120
Mile Ilić, Ljiljana Jerković
SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH
science, of the process and outcomes of teaching and learning activity and the
education process.
3. Interpret research results employing both quantitative and qualitative
approaches, by making more permanent and higher quality scientific syntheses;
4. Take into consideration the above presented methodological criteria of valuation
of the exactness of experimental research, which may help lessen the use of
pseudo-methods and the gaining of quasi-scientific results and insights in
experimental pedagogical research.
References
а ђ , . и
к њак, .
. М
л гија
аг гиј .
а
а з
а шких
ш а аЈ
а иј .
2. Cohen, L., Manion, L. & Morrison, K. (2007). Metode u istraživanja u obrazovanjuc
(prevod petog izdanja). Zagreb: Naklada Slap.
3. Fajgelj, S. (2014). Metode istraživanja ponašanja (V dopunjeno izdanje). Beograd:
Centar za primenjenu psihologiju
4. Kocić, Lj.
. Pedagoški eksperiment – karakteristike i mogućnosti. Beograd:
Institut za pedago:ka istraživanja i Prosveta.
5. Kukić, S.
. Metodologija znanstvenog istraživanja. Sarajevo: Sarajevo
publishing.
6.
а ић, .
.М
л гија на чн г а а. ања
ка Ака
ија а ка и
ј
и
ик
к .
7. McGowan, H. M. (2011). Planning a Comparative Experiment in Educational
Settings. Journal of Statistics Education, Volume 19, Number 2.
8. Milas, G. (2009). Istraživačke metode u psihologiji i drugim društvenim znanostima.
Zagreb „Naklada Slap .
9. Mužić, V.
. Metodologija pedagoškog istraživanja. Sarajevo: Zavod za
udžbenike i nastavna sredstva.
10. Savićević, D.
. Metodologija istraživanja u vaspitanju i obrazovanju. Vranje:
Univerzitet u Ni:u Učiteljski fakultet u Vranju.
11. Бe:ić, ”.
. Osnovi metodologije društvenih nauka. ”eograd Naučna knjiga.
1.
European Journal of Education Studies - Volume 3 │ Issue 1 │ 2017
121
Mile Ilić, Ljiljana Jerković
SCIENTIFIC AND METHODOLOGICAL SOUNDNESS OF EXPERIMENTS IN PEDAGOGICAL RESEARCH
Creative Commons licensing terms
Author(s) will retain the copyright of their published articles agreeing that a Creative Commons Attribution 4.0 International License (CC BY 4.0) terms
will be applied to their work. Under the terms of this license, no permission is required from the author(s) or publisher for members of the community
to copy, distribute, transmit or adapt the article content, providing a proper, prominent and unambiguous attribution to the authors in a manner that
makes clear that the materials are being reused under permission of a Creative Commons License. Views, opinions and conclusions expressed in this
research article are views, opinions and conclusions of the author(s). Open Access Publishing Group and European Journal of Education Studies shall
not be responsible or answerable for any loss, damage or liability caused in relation to/arising out of conflicts of interest, copyright violations and
inappropriate or inaccurate use of any kind content related or integrated into the research work. All the published works are meeting the Open Access
Publishing requirements and can be freely accessed, shared, modified, distributed and used in educational, commercial and non-commercial purposes
under a Creative Commons Attribution 4.0 International License (CC BY 4.0).
European Journal of Education Studies - Volume 3 │ Issue 1 │ 2017
122