Increasing Creative Output by Visually Enhancing Engineering Design Tools By David D. Harr May, 2022 Director of Thesis: Dr. Brian Sylcott Major Department: Department of Engineering ABSTRACT Innovation does not come about by random chance but is intentionally cultivated by the efforts of a designer. Many strategies exist for approaching design, ranging from the instinctual and intuitive to the more technical or analytical methods. When it comes to design, engineers are continually striving to improve the effectiveness of the design process. One area of the engineering design process deserving of attention is the ideation phase. Ideation refers to the brainstorming and idea generating activities that usually happen early in the design process. When faced with a problem engineers work to gather as many potential solutions as possible. Having a large body of initial ideas helps designers converge on an optimum final solution. Engineers have developed numerous analytical ideation tools to guide cognitive design processes and increase ideation productivity. This research investigates the effects of enhancing conceptual design tools in accordance with recommendations from the field of cognitive science. Pedagogy and learning theory literature frequently advocate for the use of multimodal representation. This refers to using multiple sensory avenues like text, sound, and visuals to communicate more effectively. A common application of this multimodal principle is to supplement text with visuals. This research investigates the impact of such a recommendation within the context of design ideation. An experiment was organized to evaluate the effect of adding visual icons to an analytical ideation tool. Using a panel of expert graders, the ideation results of engineering students were graded. This data was then statistically analyzed to look for correlations between the merit of the ideation outcomes and the presence/absence of visual icons. Ultimately, no correlation was found between increased merit in ideation outputs and the presence of visuals in the ideation tool. Upon reflection, it was proposed that there are simply other factors which had a bigger impact on the ideation results in the context of this experiment. Finally, the investigation added insight into the use of different parameters for measuring ideation effectiveness including quality, quantity, novelty, feasibility, and variety. The statistical analysis revealed that in this experiment a positive correlation existed between all five metrics. This implies that in certain applications researchers may be able to justify only using one criterion for evaluating creative ideation output. Increasing Creative Output by Visually Enhancing Engineering Design Tools A Thesis Presented To the Faculty of the Department of Engineering East Carolina University In Partial Fulfillment of the Requirements for the Degree Masters of Science in Mechanical Engineering By David D. Harr May, 2022 © David D. Harr, 2022 Increasing Creative Output by Visually Enhancing Engineering Design Tools By David D. Harr APPROVED BY: Director of Thesis Brian Sylcott, PhD Committee Member Seth Orsborn, PhD Committee Member Chris Venters, PhD Committee Member Kwaku Adu-Gyamfi, PhD Chair of the Department of Engineering Barbara Muller-Borer, PhD Dean of the Graduate School Paul J. Gemperline, PhD Table of Contents List of Tables…………………………………………………………………………..v List of Figures……………………………………………………………………...…vi 1. INTRODUCTION ................................................................................................. 1 2. BACKGROUND ................................................................................................... 2 2.1 The Function Basis ........................................................................................ 2 2.2 The Example Stimuli Strategy ....................................................................... 3 2.3 Felder-Silverman Learning Styles Model ...................................................... 4 2.4 Universal Design for Learning (UDL) ........................................................... 6 2.5 Aspects of the Learning Theories Being Studied ........................................... 7 3. LITERATURE REVIEW ...................................................................................... 8 4. THE ELEMENTS OF DESIGN IDEATION TOOL ............................................ 12 4.1 The Elements of Design Ideation Tool ........................................................ 12 4.2 Using The Elements of Design Ideation Tool .............................................. 12 5. METHODS ......................................................................................................... 14 5.1 Participants ................................................................................................. 14 5.2 Experimental Procedure .............................................................................. 14 5.3 Evaluation Method ...................................................................................... 15 6. RESULTS ........................................................................................................... 18 6.1 Expected Results ......................................................................................... 18 6.2 Inter-Rater Agreement ................................................................................. 18 6.3 Statistical Analysis Results .......................................................................... 20 7. DISCUSSION ..................................................................................................... 26 7.1 Visual Icons and Idea Fixation .................................................................... 26 7.2 The Influence of Instruction ........................................................................ 27 7.3 Discussion of Evaluation Criteria ................................................................ 28 7.4 Correlation of Evaluation Criteria ............................................................... 28 7.5 Sample Size and Participant Demographics ................................................. 29 7.6 Discussion of Other Variables ..................................................................... 30 7.7 Discussion of Further Research ................................................................... 31 8. CONCLUSION ................................................................................................... 32 References ............................................................................................................... 33 Appendix A: Elements of Design Ideation Tool ....................................................... 35 Appendix B: Transcript of Experiment Instructions ................................................. 41 List of Tables Table 1: Experimental Groups ........................................................................... 14 Table 2: Experiment Population ........................................................................ 14 Table 3: Correlation Coefficient Matrix ............................................................ 19 Table 4: Two Tailed p-value Results ................................................................. 20 List of Figures Figure 1: Example Function Decomposition [14] ................................................ 2 Figure 2: Function Basis taken from [1] .............................................................. 3 Figure 3: Dimensions of Learning and Teaching Styles [6] ................................. 6 Figure 4: TRIZ Ideation Tool [13] ..................................................................... 11 Figure 5: Multimodal Elements of Design Tool Page 1 ..................................... 12 Figure 6: Ideation Grading Rubric ..................................................................... 17 Figure 7: Inter-Rater Agreement Visualization .................................................. 18 Figure 8: Inter-Rater Agreement Graphs ........................................................... 19 Figure 9: Combine Grader Data Summary ........................................................ 21 Figure 10: Grader 1 Data Summary ................................................................... 22 Figure 11: Grader 2 Data Summary ................................................................... 23 Figure 12: Grader 3 Data Summary ................................................................... 24 Figure 13: Grader 4 Data Summary ................................................................... 25 Figure 14: Average Number of Ideas per Class Section ..................................... 27 Figure 15: Participant Demographics ................................................................ 29 1. INTRODUCTION Engineering design focuses heavily on a designer’s ability to create new, innovative, and unique solutions. An early step in the engineering design process is ideation, sometimes referred to as brainstorming. In this phase of the design process the goal is to generate a large number of ideas. Before implementing any solutions, an engineer will brainstorm as many potential solution ideas as possible. The engineer can then sort through the lists of ideas and choose the one which best addresses the problem. Investing time up front in an ideation session helps designers achieve a better overall outcome since they are able to draw from a large body of ideas. However, during ideation designers often struggle to generate a sufficient body of good ideas from which to work. To aid in the pursuit of novel ideas engineers have developed a multitude of techniques and tools for increasing creative output during brainstorming sessions. Examples of such tools include the function basis [1], the use of example stimuli [2-3], the SCAMPER method [4], Morph charts [5], and others. Analytical design tools, such as these, provide engineers with a framework to guide their thoughts. A structured ideation tool can lead a designer to make connections and generate ideas which they otherwise would not have. This study focuses on the effects of enhancing such ideation tools by applying the principles set forth by modern learning theories. Learning theories such as the Felder-Silverman Learning Styles Model [6] and Universal Design for Learning (UDL) [7] provide a framework for enhancing the way diverse groups of learners understand and communicate knowledge. Both learning theories address the topic of how to present information to maximize comprehension. One key theme which emerges is that presenting information through multiple sensory avenues increases a learner’s ability to comprehend information. A common recommendation set forth by these learning theories is to present information using both visual and verbal means whenever possible. However, most analytical engineering ideation tools are presented to students using only verbal (spoken and written) delivery. This study wonders if the usefulness of engineering ideation tools might be increased if the representation techniques proposed by learning theories were implemented. The hypothesis was, if presenting material in a multimodal way has been shown to increase student comprehension; then the same treatment of ideation tools might increase students’ ability to comprehend and implement the strategies of those tools. After all, ideation tools are simply a framework to guide higher level ideation. Perhaps multimodal representation could further clarify these frameworks to students. This led to the hypothesis that, on average, students would more effectively implement multimodal ideation tools when compared to unimodal ideation tools. This hypothesis is based on the assumption that multimodal representation increases a student’s comprehension and in turn implementation of the ideation techniques. To test this hypothesis, an analytical ideation tool was developed. The Elements of Design Ideation Tool (seen in Appendix A) combines aspects of the function basis and example stimuli strategy. In this experiment the Elements of Design Ideation Tool was presented to students using both verbal (text only) and multimodal (text and visual) delivery. Statistical analysis was conducted to determine how the presence of visual icons affected creative output. All in all, this work investigated the claim that: creative output can be improved by increasing the depth at which designers comprehend the analytical ideation tools they use, and that this improved understanding can be attained by enhancing ideation tools with the multimodal representation techniques proposed by modern learning theories. 1 2. BACKGROUND 2.1 The Function Basis The Elements of Design Ideation Tool developed in this study combines aspects of the function basis [1] and the example stimuli technique [2-3]. The function basis is an element of the engineering design technique known as function decomposition [1] [14]. In function decomposition designs are broken down into their sub-functions, and a flow diagram showing the flow of materials, signals, and energy is drawn. An example function decomposition flow diagram for the design of a mouse trap is shown in Figure 1. Figure 1: Example Function Decomposition [14] The function basis is a table typically used in conjunction with function decomposition. The function basis is organized as a tree diagram with three successive tiers. Each tier of the function basis further breaks down the broad categories of material, signal, and energy. In this way, the function basis serves as an exhaustive list of the categories of components used in engineering design. When creating a function decomposition flow chart, the function basis is used as a reference for identifying all the different elements flowing through a system. During an ideation session, however, the function basis can be used to help designers consider all the alternative methods available for addressing the problem. The function basis, taken from [1], is shown in Figure 2. 2 Figure 2: Function Basis taken from [1] One benefit of the function basis is that it uses abstract categories of design components. The secondary and tertiary levels of the function basis break down the primary three categories. This is done in such a way that it guides designers to consider a variety of subcategories, while still leaving the categories broad enough to allow for creativity. Designing at an abstract level can increase creativity and novelty since the designer is free to brainstorm ideas without being overwhelmed with technical details [15]. Additionally, by presenting the categories in a broader abstract nature, the function basis is able to present a comprehensive picture of the possible design elements. Attempting to construct a comprehensive model with concrete elements quickly becomes infeasible, considering the sheer volume of unique concrete solutions. However, specific concrete examples are not without merit. In fact, providing specific examples can also lead to increased creative output when applied to a design session correctly. For this reason, the Elements of Design Ideation Tool supplements the function basis with characteristics of the example stimuli strategy. 2.2 The Example Stimuli Strategy The example stimuli strategy refers to the practice of presenting a designer with concrete example solutions in an effort to stimulate further creativity. Exposure to example solutions can aid in the ideation process by allowing the designer to see what ideas and solutions already exist. The designer can then expand upon the existing ideas and include unique aspects from other designers’ works. In Goucher-Lambert’s study on inspirational design stimuli it was 3 observed that designers exposed to novel example solutions tended to produce more novel final designs [2]. Another benefit of example stimuli is their potential to increase feasibility. When a designer is aware of what kinds of solutions have been successfully implemented, they are able to generate ideas which can be effectively implemented in similar fashion. Ultimately, providing designers with examples of existing solutions has the potential to increase creative output by expanding the designer’s focus to consider other alternatives. The Elements of Design Ideation Tool applies the example stimuli strategy by including concrete solutions within the categories of the function basis. The example solutions demonstrate the principle of their parent category. For instance, the thermal energy category contains several concrete examples (including a burner, heat exchanger, inductor, and others) to demonstrate the different ways thermal energy has been utilized in previous applications. One negative byproduct of using example stimuli in engineering designs is a principle known as fixation. Fixation occurs when a designer subconsciously becomes constrained by the example solutions. Providing an example solution can cause designers to operate under the assumption that the only possible solutions are those based on the same fundamental principles as the example. This type of fixation can serve as a barrier to developing unconventional and novel solutions. Considering this, much research has been done in the area of engineering design to understand and overcome idea fixation. In Toh’s study comparing the effects of 2D pictorial examples verses 3D example stimuli, it was found that 3D examples can reduce novelty by causing increased fixation on specific design features [3]. Toh concluded by recommending that designers, “perform an initial round of brainstorming using pictorial examples before interacting with physical examples of existing products in an effort to broaden the solution space explored” [3]. The work asserts that the more abstract nature of pictorial examples has the potential to reduce fixation while still providing the benefits of an example stimulus. Findings such as this support the decision to include pictorial examples in the Elements of Design Ideation Tool. Another way in which the Elements of Design Ideation Tool seeks to overcome design fixation is by providing multiple examples within each category of the function basis. By including multiple examples and encouraging the designer to think of new examples, the tool suggests to the users that each category can have multiple applications. This is in contrast to providing only one example which could lead the designer to fixate on that particular example’s application. In the Elements of Design Ideation Tool, the examples were not intended to apply to any particular design problem, but instead served to further define the abstract categories of the function basis. This study sought to develop a versatile ideation tool applicable to a variety of design problems and disciplines. The concrete stimuli’s primary role within the tool is to provide examples of some of the different ways in which the sub-categories of material, signal, and energy have been applied in the past. It is thought that by viewing examples of how the categories can be applied in different ways, designers will be led to consider more unconventional approaches to solve their design problem. 2.3 Felder-Silverman Learning Styles Model Learning styles theories, such as the Felder-Silverman model, serve as a framework for educators to better understand the different ways in which learners process, organize, and understand information while learning. Learning theories such as these add value to education by clearly articulating and categorizing the different approaches used to understand information. Once learners and teachers are made aware of these learning styles, they are better 4 suited to identify which styles they have a preference for and in what areas they have room to improve. By providing deeper insight into the way people learn, learning theories have the potential to increase a learner’s aptitude for learning, no matter the subject. One common point of contention surrounding the learning styles theory is the meshing hypothesis. The meshing hypothesis asserts that matching the teaching style with the student’s learning style will yield the best learning outcomes. Several studies have conducted tests and found no evidence in support of the meshing hypothesis [16-17]. Studies such as these bring to light some of the common misconceptions surrounding the study of learning styles. Felder identifies the three most common misuses of the learning styles theory to be the belief that learning styles are (1) strictly either-or categories, (2) invariant over time and place, and (3) an inherent measure of a learner’s strengths and weakness [18]. Attempting to use learning styles theory under these false assumptions leads to several problems for instructors. When introduced to learning styles theory, many instructors are concerned with the feasibility of attempting to match every student in a class with a unique instruction method. Furthermore, certain lessons and concepts can objectively be argued to be best taught using a particular style. Should instructors attempt to use an inferior method of teaching simply to match with students who prefer that style? Surely not. In fact, the best practice for applying the learning styles theory does not focus on attempting to match each student with one particular instruction style. Instead, teachers should use an instructional method which delivers on all elements of the learning styles framework. In this way lessons are delivered holistically and easily understood no matter which style the learner chooses to approach the lesson. Learning styles can perhaps be more accurately described as learning preferences [6]. While a learner may prefer a particular style and lean towards it more often, this does not mean the learner is incapable or unwilling to use other styles. In fact, many learners will choose to apply different stylistic techniques depending on the subject, material, and context. Additionally, learners’ skills at applying the different elements of learning styles are not stagnate. With practice learners can grow and improve their ability to apply different stylistic approaches. All in all, a learner’s stylistic preferences and their proficiency at applying them will change across time and setting. Understanding this leads to a more useful application of the learning styles theory. Once learning styles are viewed as an organized list of the different ways in which people can learn, effective teachers and learners will seek to become proficient across all categories. 5 The Felder-Silverman model lays out five dimensions of teaching/learning styles with two stylistic preferences in each dimension, see Figure 3. Figure 3: Dimensions of Learning and Teaching Styles [6] Felder asserts that traditional engineering education techniques cover half of these learning styles (intuitive, auditory, deductive, reflective, and sequential) [6]. This observation holds true when examining current ideation tools. The function basis and SCAMPER methods, for example, are both presented to students using only verbal (spoken and written) delivery. Felder proposes that engineering education can be improved by utilizing teaching techniques which cover both sides of each learning dimension [6]. Considering that many engineering ideation tools only make use of verbal elements, this study focuses on the effects of enhancing ideation tools with visual elements. This study focuses primarily on the input/presentation dimension of Felder’s learning styles matrix, choosing it as the experiment’s independent variable. It is hypothesized that by presenting the Elements of Design Ideation Tool both verbally and visually, students will gain better comprehension of the tool, resulting in increased creative output. 2.4 Universal Design for Learning (UDL) Another learning theory which advocates for presenting information using multiple sensory avenues is universal design for learning or UDL. Universal design for learning focuses on making learning accessible to all learners. UDL emphasizes designing a learning environment which accommodates learners of all ability levels and learning preferences. UDL identifies three categories for designing an effective learning environment: engagement, representation, and action/expression [7]. UDL further describes engagement as the “why of learning,” representation as the “what of learning,” and action/expression as the “how of learning” [7]. The primary goal of UDL is to provide multiple means of delivery across each of these three categories. This includes implementing different ways to engage learners, represent information, and have learners process knowledge. The aspect of the UDL framework which most directly applies to this experiment is the representation category. UDL provides a vast bank of ideas for how information can be presented to learners using multiple sensory avenues. Some UDL recommendations include making use of text-to-speech technology, providing physical models, using tactile vibrations as signals, and many others. One UDL recommendation in particular, “Provide descriptions (text 6 or spoken) for all images, graphics, video, or animations” [7], serves as this study’s variable of interest. The UDL framework suggests that by presenting information with multiple mediums, like text and graphics, the information is made more accessible to a wider population of learners. “Providing alternatives—especially illustrations, simulations, images or interactive graphics—can make the information in text more comprehensible for any learner and accessible for some who would find it completely inaccessible in text.” [7]. This evidence shows how UDL supports the decision to supplement text heavy engineering ideation tools with visual icons. Universal design for learning provides numerous recommendations on how to present material in a holistic fashion. This includes providing information in a digitally customizable format. Allowing users to manipulate font sizes, colors, display windows, and visual layout increases accessibility by allowing for greater personalization. Recommendations such as these could be applied to engineering ideation tools to further increase their usability. In this study, however, only one variation to the tool will be examined in order to maintain the controlled nature of the experiment. The independent variable examined in this experiment will be the presence / absence of supporting visual icons. It should be recognized that implementing many of the other UDL techniques has the potential to further increase the effectiveness of ideation tools. As such, many of the other educational strategies set forth by UDL have the potential to serve as the basis for other similar research. Even so, this study maintains a focus on the effects of adding visual enhancement to text. 2.5 Aspects of the Learning Theories Being Studied This study focuses on the presentation and representation aspects of the learning theories discussed. More specifically, the effects of visually enhancing traditionally text dominate ideation tools will be investigated. It is acknowledged that the learning theories discussed in this work encompass a broader spectrum of recommendations for enhancing educational settings. For example, both learning theories provide recommendations for including tactile and active learning elements. Suggestions such as these can also be applied to engineering ideation tools to great effect. Another widely used medium of design tools are design card decks [19]. These tools provide designers with a deck of cards containing engineering design elements. A tool such as this allows for increased active learning and tactile engagement since the user can sort, organize, and move cards. Future research could be conducted to compare active design tools, like design cards, with tools having a more passive / reflective nature. An investigation of this nature would use a similar theoretical basis to the current work. However, drawing conclusions on this matter is outside the scope of this particular study. Ultimately, this study chose to focus specifically on the elements of the learning theories which recommended supplementing verbal text with visual icons whenever possible. 7 3. LITERATURE REVIEW The following section provides context for the thesis through a review of pedagogy and cognitive science literature. This literature review is included to add clarity and depth to the theoretical framework which supports the study’s hypothesis. This review contains examinations of similar studies and discussion on how this experiment addresses gaps and adds value to the research. In Watkins’ dissertation “Sequential Rhetoric: Teaching Comics as Visual Rhetoric” he investigates and advocates for the use of comics to teach visual literacy in technical communication [8]. The study takes a qualitative approach, focusing on students’ perceptions towards using comics in professional and academic communication. Watkins’ work is similar to this thesis in that it adds visual elements to enhance verbal/text dominate content. Watkins even makes the following statement in reference to STEM education, “That said, research looking into how much reading comprehension changes from teaching with comics vs. traditional texts for those topics would be extremely valuable. Perhaps this could be done in the future in a broader project crossing various academic borders” [8]. This thesis research does exactly that, applying the idea of visual writing enhancements to the field of engineering design. One major way in which this thesis expands and adds value to Watkins’ discussion of the topic is by taking a quantitative statistical approach, since Watkin’s research is heavily qualitative. A statistical research approach can add further credibility and support to the hypothesis, as well as provide numerically measurable results. A reoccurring idea from the education literature was the recommendation to give students the opportunity to construct their own visual representations of the material [8-9]. This technique has found success at increasing comprehension since students must gain a detailed understanding of content in order to create an accurate visual representation. One way in which this technique can be applied to the Elements of Design Ideation Tool is to ask students to come up with their own example icons for each category. The exercise of brainstorming new examples for each category leads the designer to further explore the design space. Further support for the hypothesis that visually enhancing text improves communication and understanding comes from the field of cognitive science. In Holden’s paper “Visuals In E-Learning: What We’ve Learned from Cognitive Science” [9] he supports the use of multimodal presentation and brings forward other researchers’ works in support. Holden’s work asserts that presenting material in a multimodal fashion is in general superior to only using one mode of representation. Holden cites many works including (Enns, 2004) who concludes that students will learn more from instruction which appropriately integrates text and visuals. This thesis’ research expands on the platform of cognitive science research by investigating whether students can produce superior ideation results when relying on multimodal conceptual design tools. The assumption guiding this hypothesis is that if multimodal delivery increases learning comprehension, then a multimodal design tool should be easier for students to comprehend. The better students can understand the design tool; the easier it will be for them to use that tool to increase their creative output during ideation. Holden’s work also introduced several key ideas from the study of cognitive science [9]: 8 1. Dual Coding Theory – The idea that the brain has separate channels for visual and verbal information, and that working memory can be increased by providing information through both channels. This theory is one of the fundamental ideas underlining this study’s hypothesis. 2. Cognitive Flexibility Theory – Emphasizes the importance of giving learners the opportunity to construct multiple representations of the material. 3. Cognitive Load Theory – Focuses on not overwhelming the working memory with too many inputs. The Cognitive Load Theory cautions against excessive multimodal instruction as this could result in cognitive overload. Mayer and Moreno address this topic in detail and discuss strategies to reduce cognitive load while effectively using multimodal instruction [10]. The following list summarizes the main ideas and relevant recommendations from [10]. 1. Use Multiple Input Channels: Each of the five senses works as an input for information. The more inputs that are engaged the easier it is to comprehend the information. 2. Segmenting Material to Increase Processing Time: Provide information in segments and allow learners time to process between each segment. While presenting information in segments does require additional time during the initial lesson, it saves time overall since students are more likely to understand the material the first time. 3. Focus on Important Information: Eliminate distractions and unnecessary information. Signal / highlight important information. 4. Put Relevant Material Together: Eliminate unnecessary processing by organizing material so that related content is intuitively linked. For example, if text is used to explain a visual then the two should be placed next to each other. 5. Avoid Simultaneously Sending Different Messages Through the Same Channel: For example, talking to someone while they are reading attempts to simultaneously put two different inputs (spoken words and written words) through the verbal channel. This is distracting and the receiving individual will only be able to focus on one message at a time. However, if the individual speaking is dictating the same words that the reader is visually reading this is advantageous since the reader is simultaneously receiving the same message through the eyes and ears. Another interesting example is closed caption videos. It requires more cognitive processing power to watch a silent, closed-captioned video since the eyes attempts to rapidly bounce between the text captions and visual images. The ideal format for balancing simultaneous visual and verbal information is to watch a narrated video. In this way the verbal information is received through audible words and the visual information through the eyes. The following section reflects on how the above listed techniques apply to the multimodal Elements of Design Ideation Tool analyzed in this study. 1. Use Multiple Input Channels: One of the primary suggestions from this paper was to make use of simultaneous auditory and visual delivery. While the Elements of Design Ideation Tool does not have an auditory element built in; this is definitely a technique that will be implemented when providing instruction on how to use the tool. 2. Segmenting Material to Increase Processing Time: When delivering instructions on how to use the tool, the tree diagram portion of the table was introduced first. After 9 allowing students to process the first half, the examples section of the tool was introduced and explained. Additionally, students were allowed processing time to reflect and study the Elements of Design Ideation Tool prior to beginning the ideation session. 3. Focus on Important Information: The original function basis which appeared in [1] had a third column. This column was removed when adapting the Elements of Design Ideation Tool since it was unnecessary information when applying the table as a conceptual design tool. This third level was replaced with the more relevant and applicable examples section. 4. Put Relevant Material Together: By placing the text descriptions immediately below the icons they are intuitively linked. Additionally, every other row of the Elements of Design Ideation Tool is shaded, making it easy to tell where each element falls in the tree structure hierarchy. 5. Avoid Simultaneously Sending Different Messages Through the Same Channel: In the Elements of Design Ideation Tool both the icons and text are received through the eyes. From a theoretical point of view this would typically be unadvisable. However, the difference is that the Elements of Design Ideation Tool does not attempt to push both pieces of information through the eyes at the same time. The reader can simply read the text, then look at the icon, and associate the two. This strategy does require representational holding, which [10] refers to as the need to delegate cognitive processing power to hold one piece of information in working memory while receiving the second piece of information. Even so, it is reasonable to assume that the amount of cognitive processing power required to hold one element of text while relating it to a single icon is minimal. Initially, there was discussion of adding an additional experimental group to the study where the Elements of Design Ideation Tool contained only icons with no text descriptions. After considering Haramundanis article “Why Icons Cannot Stand Alone” [11] it was decided that including such an experimental group would hold little value since the tool has no reference for the reader to understand the icons’ meanings. Haramundanis discusses how icons by themselves hold little value unless the icon is accompanied by a text description or the reader has prior knowledge of the icon’s meaning. The benefit of using an icon is that it provides a quicker reference once the reader knows its meaning. If designed well, icons can clarify and add context to a word or description. Another interesting study investigated the effectiveness of icons for communication by evaluating different visual representations of a medication schedule [12]. The study found less success when using unique clock like visual schedules when compared to a more traditional timeline visual. From these finding the study concluded that “compactness and familiarity are important for designing icons” [12]. Furthermore, the results showed that the text schedule led to better recall and understanding than the visual representations used. Even so, the article pointed out that much research exists to support the idea that using icons and text in conjunction could lead to optimal results. Where one medium might be lacking, the other could complement it and add clarity. Overall, the research concluded by stating that icons hold potential value for representing schedule information, but that additional training may be required for users to understand complex or unfamiliar icons. The article also comments on the cognitive load theory with the statement, “flowcharts are less effective than text instructions for diagnosing computer problems when multi-page charts are used, suggesting that benefits of the visual icon format can be reduced when information must 10 be integrated across pages (Desaulniers et al, 1988)” [12]. The Elements of Design Ideation Tool is too large to readably fit on a single 8.5 x 11 page. The tool was broken into 3 pages according to the materials, signals, and energy sections. There is the potential that these page breaks could introduce unnecessary cognitive load. On the other hand, the amount of information that must be integrated across pages is minimized since each page break corresponds to one of the three primary sections. In fact, some designers may find it easier to focus on one section of the tool at a time. Furthermore, the pages are single sided and unbound such that they can be spread out on a table and viewed together. Lastly, this literature review concludes by pointing out how using icons to enhance ideation tools is already an idea which has emerged. As seen in Figure 4 where the TRIZ (Theory of Inventive Problem Solving) ideation tool uses images to clarify its ideation prompts. This study adds value to this area of engineering design by investigating the effect these icons have on creative output during ideation. Figure 4: TRIZ Ideation Tool [13] 11 4. THE ELEMENTS OF DESIGN IDEATION TOOL 4.1 The Elements of Design Ideation Tool This study focuses primarily on the presentation/representation aspects of the learning theories. As such, the presence of visual and verbal elements in the tool serves as the experiment’s independent variable. One version of the tool contains only verbal elements (written words) while the other contains both verbal and visual elements. The two versions of the design tool can be seen in Appendix A. An image of the first page of the multimodal tool is provided in Figure 5. While the experiment will vary only one aspect of the learning theories, the Elements of Design Ideation Tool makes use of many of the learning theories’ other key aspects. For example, the Elements of Design Ideation Tool’s combination of abstract categories from the function basis and concrete examples from the example stimuli strategy embrace both sides of the content dimension of Felder’s theory. Furthermore, the layout of the tool lends itself to both sequential and global learning preferences. The tool can be used by working through each row and tier sequentially, similar to the analytical approaches employed by the SCAMPER [4] and Morphological [5] methods, or globally by jumping from one area of the diagram to another as needed. In summary, while the primary focus of this study is on the visual-verbal aspect of the learning theories, the Elements of Design Ideation Tool was built with the complete framework of the learning theories in mind. Figure 5: Multimodal Elements of Design Tool Page 1 4.2 Using The Elements of Design Ideation Tool The following section provides instructions and examples of how the Elements of Design Ideation Tool is intended to be used. Participants in the study received similar instruction prior to using the tool. 12 The Elements of Design Ideation Tool is intended to serve as a catalog of design elements. In engineering design there are three categories of design elements: material, signal, and energy. The tool organizes these three categories using a tree diagram that further breaks down each of the three primary categories. To effectively utilize the tool, designers should consider all the different elements of design and how they apply to the specific design problem. Designers should ask themselves questions such as: What different types of materials exist that could be used to solve the problem? What different methods exist for transferring information and signals? What are the different types of energy that can be used to generate and supply power? Most importantly, how can each of these concepts be applied to the specific design problem? One method for using the tool is to use a systematic approach. The designer may know that their solution requires a source of input energy. To generate a large body of ideas the design can sequentially work through the energy section of the Elements of Design Ideation Tool. In this case, the designer would first come across the category of human energy. They would then ask themselves, what are all the different ways human energy can be implemented to solve the problem? The designer would then move to the next section, asking the same question about acoustic energy, and so on. This method is useful for generating a large quantity of ideas. It also forces the designer to consider unconventional categories, which can lead to novel solutions. Additionally, the Elements of Design Ideation Tool contains concrete examples. The tool includes specific examples within each of the categories. It should be noted that these examples are by no means comprehensive. Instead, the examples serve to increase the designer’s understanding of each category. The six examples included in each category were chosen in an attempt to represent a wide span of concepts and applications within each category. An attempt was made to intentionally include unconventional examples in the hope that it would lead designers to more unique ideas. When used effectively, the examples will broaden the designer’s perception of how each category can be employed. For example, when the designer first considers the thermal energy category they may think of fire. However, when seeing an inductor listed as an example, the designer will be led to consider electricity as a means for supplying thermal energy. Additionally, designers should be instructed to think of their own examples for each category. The process of generating new examples will push the designer to further mentally explore each category. The layout of the Elements of Design Ideation Tool as a tree diagram, moving from broad categories to specific examples, is intended to encourage designers to work up and down the tree throughout the ideation process. The designer should spend time generating ideas at both the broad/abstract level, as well as considering the more specific categories. 13 5. METHODS The experiment’s objective is to evaluate how adding visuals to an engineering design tool affects its usefulness for generating creative ideas. 5.1 Participants The participants selected for this study were students in East Carolina University’s sophomore level engineering design course: ENGR-2000. Similar studies in engineering design literature have also chosen sophomore students in a design course as participants [15]. The study was conducted in the Fall 2021 semester and contained 60 participants across 3 sections of the course. This course was selected as the population for the study since the course contained only students majoring in engineering, and the design focused curriculum of the course matched well with the topic of this study. Only students enrolled in their second design course, ENGR-2000, were used since it was desired that all participants be at an approximately equal education and experience level. For completeness, students were asked to provide the following demographic information: age, gender, ethnicity, GPA, engineering concentration, and answer the question “Is this the first time you have used this style of engineering ideation tool?” 5.2 Experimental Procedure Participants were divided into two experimental groups. Each experimental group received a different version of the design tool to use during the ideation session. The two experimental groups are listed in Table 1. Table 1: Experimental Groups Group 1 Ideation Tool with only Text Elements (Verbal) Group 2 Ideation Tool with both Text and Visual Elements (Multimodal) *Examples of each version of the Elements of Design Ideation Tool are provided in Appendix A. The study involved 60 participants. These participants were distributed across three sections of the course with 18 students in the first class, 18 students in the second class, and 24 students in the third class. As seen in Table 2, each of these classes was led by a different instructor. It was decided that Instructor 1 and Instructor 2’s classes would be used as the two experimental groups compared in the statistical analysis. This decision was made since these two class sections had an equal number of students and because Instructor 1 and Instructor 2 used the same curriculum schedule, whereas Instructor 3 used a slightly different course layout. The data from Instructor 3’s class was used for practice during the grader norming session. Table 2: Experiment Population Instructor Students Experimental Group Data Used Instructor 1 18 Group 1 (Verbal) Yes, used in statistical analysis Instructor 2 18 Group 2 (Multimodal) Yes, used in statistical analysis Instructor 3 24 Group 2 (Multimodal) No, used in grader norming session Prior to conducting the study, participants were given a brief introduction to the study’s topic and methods. Additionally, the participants were instructed on how to use the Elements of 14 Design Ideation Tool. The instructions consisted of a conceptual overview of the tool as well as examples of the tool being used. Examples were provided since in Nagel’s study [15] it was found that engineering students demonstrated better retention and application of skills when instructed using both lecture and examples, as opposed to just lecture. Furthermore, in accordance with the recommendations of UDL [7], the instructions were delivered using both spoken word and text/visuals where applicable. The instructions were presented using a prerecorded video to avoid introducing variability in the way the instructions were presented. Across the experimental groups, the only difference in the instructional videos was which version of the Elements of Design Ideation Tool appeared when giving examples of how to use the tool. After receiving their instructions, participants were asked to take 5 minutes to familiarize themselves with the Elements of Design Ideation Tool before beginning the ideation session. Participants then began a 30 minute ideation session. In the ideation session participants were provided a problem statement and asked to use the Elements of Design Ideation Tool to generate a list of possible solutions to the problem. All participants were provided the same problem statement, such that the only variable was which version of the tool the participants used. The problem statement used in this study was modeled after the problem statement used in Goucher-Lambert’s study [2]. This problem statement addresses a topic familiar enough to the participants such that additional research was not required to understand the problem, and yet broad enough to allow room for creative solutions. Problem Statement: Many individuals operate a cell phone while walking. The problem is, they walk into obstacles and people since they are not looking where they are walking. Participants submitted their lists of ideas electronically using the Canvas Learning Management System. 5.3 Evaluation Method What is creativity? Different sources will provide different definitions and explanation as to why something may be considered more or less creative. In this study, the lists of ideas generated by the participants were scored by engineering faculty members knowledgeable in the area of engineering ideation. The lists of ideas were scored across five categories: quantity, quality, feasibility, novelty, and variety. These metrics are commonly used in engineering design literature to evaluate the productivity of an ideation session [2] [20]. At first glance some of these criteria seem to address the same idea. For this reason, it was critical to establish clear definitions for each. The faculty graders spent time discussing and clarifying the definitions of each criterion. This was done to ensure that the grading in each category reflected a specific attribute of creativity. Figure 6 shows an example of the rubric which was used to evaluate the effectiveness of participants’ ideation sessions. Prior to having the faculty graders conduct their independent evaluations, a group norming session was conducted. In this session, all the graders discussed the rubric and practiced evaluating student work. This was done to help the graders reach a collective understanding of the evaluation criteria and reduce the subjective nature of the grading method. During the grader norming session, the four faculty graders collaborated to clarify and edit the definitions of each evaluation metric. Detailed discussion was given to understanding the key differences between each criterion. The graders came away from the session with a consistent 15 understanding of how to grade students’ work in each category. The following definitions were used to understand the key differences between each evaluation criteria. ? Quantity, also sometimes referred to a fluency, is generating a sufficient number of ideas during ideation. In ideation it is understood that more ideas are better for problem solving. ? Quality is how well a solution addresses the particular problem statement given. An idea might have high novelty, variety, and feasibility; but if it does not pertain to the specific problem statement given, then the idea has low quality with respect to the project. ? Feasibility or usefulness pertains to the idea’s technological practicality. A highly feasible idea is one which can reasonably be implemented given the available technology and constraints. ? Novelty is how unique an idea is compared to existing solutions. For an idea to be considered novel it must introduce a new approach for solving the problem, as compared to the current solutions used to address the issue. ? Variety, sometimes called flexibility, is the scope of ideas represented. This refers to the number of different fundamental categories or disciplines represented in the idea set. If all of the ideas are based on the same fundamental principle then the student would score low in variety. If the ideas use a wide spectrum of fundamental principles then the student should score highly in variety. 16 Student Number 1 Quantity (Fluency) Was a sufficiency number of ideas generated? Quality How well do the ideas address the problem statement? Feasibility (Usefulness) How technologically feasible are the ideas? Can it actually be done within reasonable constraints. Novelty How unique are the ideas compared to existing solutions? Variety (Flexibility) Scale How widely spread is the scope of ideas? How many categories of ideas are represented? Excellent 9-10 If all of the ideas are based on the same fundamental principle then the Good 7-8 student should score low in this category. If the idea makes use of a wide Average 5-6 variety of fundamental principle then the student should score highly. Poor 3-4 Total Score 0 Failure 1-2 Figure 6: Ideation Grading Rubric This study used four faculty graders. Each student’s list of ideation results was independently graded by each of the four graders. The statistical analysis compares the scores given by each individual grader and the average of all four graders combined. 17 6. RESULTS 6.1 Expected Results Based on cognitive science, pedagogy, and learning theories literature it was expected that participants who had access to the multimodal tool (group 2) would perform better than participants using the only verbal tool (group 1). However, the results found during this experiment do not unequivocally agree with this hypothesis. 6.2 Inter-Rater Agreement It should first be recognized that the evaluation method used in this experiment is subjective in nature. Each faculty grader likely has different opinions on what constitutes excellent ideation results. Even so, using a panel of expert graders is necessitated by the subject matter. Evaluating the creative merit of ideas to an open-ended problem statement is not something that has a definitive right or wrong answer. In an effort to eliminate as much ambiguity as possible, the faculty graders spent time discussing and revising the grading rubric, such that all graders had a clear and unified understanding of the criteria. Considering this, it is worth investigating the level of inter-rater agreement demonstrated by the results. To visualize this the total score given to each student by the four faculty graders is graphed in Figure 7. Figure 7 can be used to determine if the graders tend to agree about each student. A strong inter-rater agreement is characterized by the graders’ trend lines moving up and down together. It is worth noting that one grader may tend to give higher scores overall while another tends to give lower scores, but the graders can still be said to agree if the relative up and down movements are similar. Max Score Min Score Range Standard Deviation Mean Score Grader 1 44 12 32 7.31 30.25 Grader 2 44 11 33 9.71 25.75 Grader 3 41 10 31 6.73 26.31 Grader 4 46 5 41 11.05 16.92 Figure 7: Inter-Rater Agreement Visualization In addition, the level of inter-rater agreement is represented by the correlation coefficients documented in Table 3. The correlation coefficients in Table 3 represent how much each faculty grader tends to agree with one another. The coefficients range on a scale from 0 to 1 18 with 1 being perfect agreement and 0 being perfect disagreement. It can be seen that all combinations of graders show a correlation of at least 0.6 or better. This suggests that the trends represented in each graders' data tend to agree. Table 3: Correlation Coefficient Matrix Correlation Coefficients Grader 1 Grader 2 Grader 3 Grader 4 Grader 1 1.000 0.667 0.652 0.715 Grader 2 - 1.000 0.800 0.886 Grader 3 - - 1.000 0.864 Grader 4 - - - 1.000 Additionally, the inter-rater agreement plots with respect to each of the five grading metrics are provided in Figure 8 along with the total scores plot from Figure 7. Figure 8: Inter-Rater Agreement Graphs 19 6.3 Statistical Analysis Results The two experimental groups were compared using box-and-whisker plots for visualization and t-tests to determine statistical significance. The results were analyzed both individually and as a combined average of the faculty graders’ scores. To determine statistical significance, the two-tailed p-value was calculated for each criterion of the grading rubric as well as the total score. This p-value was then compared against a significance level (?) to determine if the observed difference in mean scores between the two groups was statistically significant. For this study, the significance level was set as ? = 0.05, which corresponds to a 95% confidence interval. If a p-value is found to be less than ?, then the difference in mean scores between the two groups is said to be statistically significant. All the p-value results are documented in Table 4. Table 4: Two Tailed p-value Results p-values If the p-value is less than an alpha of 0.05 then the t-test is considered statistically significant. These cells are highlighed in green bellow. Combined Data Grader 1 Grader 2 Grader 3 Grader 4 Total Score 0.0304 0.1409 0.0163 0.0036 0.0087 Quantity 0.0043 0.0041 0.0068 0.0073 0.0049 Quality 0.2496 0.9434 0.0378 0.2551 0.0052 Fesability 0.7714 1.0000 0.0196 0.1262 0.0488 Novelty 0.4312 0.4729 0.0457 0.1380 0.0312 Variety 0.1479 0.6535 0.0311 0.0057 0.0704 From Table 4, some key observations are that all of Grader 2’s data and nearly all of Grader 4’s data show a statistically significant difference in mean scores between the experimental groups. Of the four graders, Grader 2 and Grader 3 also have the largest Range and Standard Deviations in their grading. This could be an indication that these two graders made better use of the full 1 to 10 grading scale, and thus, their results show greater definition. In addition, all graders and the combined average show a significant difference in Quantity. Similarly, a significant difference in the Total Score is observed by all graders except Grader 1. The box-and-whisker plots for the combine data are provided in Figure 9, and the data for each grader is presented in Figures 10 - 13. Statistically significant results, as determined by the t-test, are marked with a green dot to the left of the title. 20 Combine Grader Data Summary 31 Tota l Scor2e2s.41667 30.375 41 Quantit3y0 .2S5core27s.25 25.41667 24.25 29.58333 23.75 43.25 23.5 26.58333 28.75 50 10 45 9 40 8 35 7 30 6 25 5 20 4 15 3 10 2 5 1 0 0 Verbal Multimodal Verbal Multimodal Quality Scores Feasibility Scores 10 10 9 9 8 8 87.5 5.5 4.25 7 9.75 5.25 4.75 2.75 2 3.25 2.25 9.75 2 3.25 4.75 6 6 5 5 4 4 3 3 2 2 1 1 0 0 Verbal Multimodal Verbal Multimodal Novelty Scores Variety Scores 10 10 9 9 8 8 7 7 6 6 5 5 4 4 6.5 4.333333 7 8.75 6.25 4.5 4.666667 3 6.5 7.25 5.5 8.75 53.5 6.5 6.5 2 2 1 1 0 0 Verbal Multimodal Verbal Multimodal Figure 9: Combine Grader Data Summary 21 Scores Scores Scores Scores Scores Scores ProfG. rRaedeedr 1D Sautam Smuamrym ary 30 Tota l Scores 12 30 41 Quantity 2S8cores30 29 28 31 30 42 17 37 30 50 10 45 9 40 8 35 7 30 6 25 5 20 4 15 3 10 2 5 1 0 0 Verbal Multimodal Verbal Multimodal Quality Scores Feasibility Scores 10 10 9 9 8 8 710 6 5 7 10 5 4 2 2 3 2 10 2 4 5 6 6 5 5 4 4 3 3 2 2 1 1 0 0 Verbal Multimodal Verbal Multimodal Novelty Scores Variety Scores 10 10 9 9 8 8 7 7 6 6 5 5 4 4 4 1 6 10 4 2 3 3 6 8 7 8 32 7 4 2 2 1 1 0 0 Verbal Multimodal Verbal Multimodal Figure 10: Grader 1 Data Summary 22 Scores Scores Scores Scores Scores Scores Dr. GGruaadner D 2a Stuam Smumarmy ary 30 Tota l Scores 31 27 42 Quantity 3S1cores25 18 15 22 17 44 18 26 29 50 10 45 9 40 8 35 7 30 6 25 5 20 4 15 3 10 2 5 1 0 0 Verbal Multimodal Verbal Multimodal Quality Scores Feasibility Scores 10 10 9 9 8 8 77 6 5 7 9 6 5 3 2 4 3 9 2 4 6 6 6 5 5 4 4 3 3 2 2 1 1 0 0 Verbal Multimodal Verbal Multimodal Novelty Scores Variety Scores 10 10 9 9 8 8 7 7 6 6 5 5 4 4 5 6 5 8 6 4 4 3 3 4 3 9 33 5 5 2 2 1 1 0 0 Verbal Multimodal Verbal Multimodal Figure 11: Grader 2 Data Summary 23 Scores Scores Scores Scores Scores Scores ProfG. Braodreerr 3D Sautam Smuamrym ary 35 Tota l Scores 29 28 38 Quantity 2S9cores27 25 22 29 21 41 22 17 25 50 10 45 9 40 8 35 7 30 6 25 5 20 4 15 3 10 2 5 1 0 0 Verbal Multimodal Verbal Multimodal Quality Scores Feasibility Scores 10 10 9 9 8 8 77 4 3 7 10 5 5 3 2 3 2 10 2 2 3 6 6 5 5 4 4 3 3 2 2 1 1 0 0 Verbal Multimodal Verbal Multimodal Novelty Scores Variety Scores 10 10 9 9 8 8 7 7 6 6 5 5 4 4 7 7 7 7 7 6 5 3 7 7 7 8 37 4 7 2 2 1 1 0 0 Verbal Multimodal Verbal Multimodal Figure 12: Grader 3 Data Summary 24 Scores Scores Scores Scores Scores Scores RGeriasd Dear t4a SSuummmmaaryry 29 Tota l Scores 13 17 43 Quantity 1S9cores16 11 8 13 7 46 9 10 18 50 10 45 9 40 8 35 7 30 6 25 5 20 4 15 3 10 2 5 1 0 0 Verbal Multimodal Verbal Multimodal Quality Scores Feasibility Scores 10 10 9 9 8 8 710 6 4 7 10 5 5 3 2 3 2 10 2 3 5 6 6 5 5 4 4 3 3 2 2 1 1 0 0 Verbal Multimodal Verbal Multimodal Novelty Scores Variety Scores 10 10 9 9 8 8 7 7 6 6 5 5 4 4 10 2 4 10 4 3 2 3 2 3 1 10 32 3 5 2 2 1 1 0 0 Verbal Multimodal Verbal Multimodal Figure 13: Grader 4 Data Summary 25 Scores Scores Scores Scores Scores Scores 7. DISCUSSION One of the most notable observations from this experiment’s findings was that in every statistically significant comparison, the verbal group’s mean score is higher than the multimodal group’s. This is in direct contradiction to the expected results. The following sections will discuss these results, the influencing factors, and potential causes for the recorded observations. 7.1 Visual Icons and Idea Fixation As established in the literature, one major application of icons is to clarify text captions. In situations where text may be unclear or incomplete, the right visual can significantly increase a viewer's understanding. Visuals serve to establish concepts in a concrete and more tangible way. This characteristic of icons may have defeated some of the intent behind the Elements of Design Ideation Tool. The tool featured both concrete examples and abstract categories to help designer navigate the design space. The intent was that designers would use both the concrete and abstract elements to expand the domain of their brainstorming. It is possible that the addition of icons may have had the opposite effect, with the visuals unnecessarily constraining the design space. The first three columns of the Elements of Design Ideation Tool contain the abstract design categories organized as a tree diagram. For designers to utilize this section of the tool to its fullest, they were encouraged to consider how a variety of broad design related categories might apply to their specific problem. It is critical for the designer to understand that these category titles represent broad concepts with numerous applications underneath. The addition of icons to these sections may have clarified the categories too much, making it difficult for the designer to apply the concept to their specific problem statement. For example, one category under the energy section is mechanical energy. The icon used to illustrate this was a robotic arm. Upon seeing this, it is possible that a designer would make the subconscious assumption that robotics is the only type of mechanical energy. In fact, there are many other ways in which mechanical energy can manifest, some of which are provided in the accompanying examples section. The presence of this robotic arm icon next to the abstract category heading may have constrained that category in the designer’s mind. Resulting in the designer underutilizing a significant portion of the category. One potential solution to address this issue would be to only use icons in the concrete examples section. By leaving the abstract headings as simply text they may be perceived by the designer as more ambiguous. This ambiguity may be exactly what is needed for the title of an abstract category. Allowing the designer’s mind to explore more freely. Another implication of using icons is their effect on idea fixation. One of the biggest traps with using concrete examples in the early conceptual design phase is that it can cause designers to get stuck only thinking of ideas similar to the examples they were shown. In an effort to overcome this, the Elements of Design Ideation Tool presented the user with a number of different examples. The intent was that designers would benefit from the example stimuli while still having a constant reminder that there are numerous potential applications of the concepts. Another way the study sought to reduce idea fixation was by emphasizing the point during the instruction session. The need to think outside the box and generate one's own examples was a key point addressed in the instruction. Even though attempts were made to reduce it, the effect that icons have on idea fixation should not be ignored. Visuals tend to be perceived as more concrete compared to text. This is perhaps because the icon actually shows the physical object 26 whereas a word only brings up the idea of that object. This attribute of visual icons has the potential to increase the problem of idea fixation in the examples section. When a designer sees the physical item, it is easier for them to latch onto the idea and become fixated on that concept. 7.2 The Influence of Instruction Another insightful observation which came out of this experiment attests to the observable effect that instruction has on ideation results. Perhaps even more important than the ideation tool itself is the way designers are trained to use it. When studying the results from the three class sections, a clear difference was observed in one of the classes. On average, students from Instructor 3’s class had notably longer lists of ideas compared to either Instructor 1’s or Instructor 2’s. The number of ideas on each participant’s list was counted and the average number of ideas in each class is graphed in Figure 14. The most likely explanation for this observation is the difference in approach used by the instructors. It is known that Instructor 3 strongly emphasizes the importance of quantity when instructing students on engineering ideation. It is logical that students from his class would reflect this influence in their work. This is one of the reasons why it was decided to use Instructor 3’s class as a sample for the grader norming session and compare Instructor 1’s and Instructor 2’s classes in the statistical analysis, since the instructional design in those classes was most similar. Figure 14: Average Number of Ideas per Class Section The instructor’s effect on the results was anticipated, and the experimental procedure sought to reduce the influence of varying instruction by using the same prerecorded instructions to teach each class about the tool. However, the prerecorded video was only one lesson in a larger course on engineering design. The overarching influence each individual instructor’s style would have on their class cannot be ignored. Even though this one particular lesson was delivered with identical instruction, the larger influence of each individual instructor throughout the course would have a significant impact on students’ overall approach to conceptual design. Ultimately, the key takeaway from this observation is that the type of instruction and areas focused on during ideation training has a direct effect on the output produced by a designer. Furthermore, one cannot only consider the instructions given in relation to one specific project, as a designer’s history and the context will affect a designer's approach to ideation. 27 7.3 Discussion of Evaluation Criteria In scholarly research the concept of creativity is typically defined using a set of sub-criteria which combine to address overall creativity. Looking at the evaluation criteria used in other research, this study identified five metrics for evaluating creativity: Quantity, Quality, Feasibility, Novelty, and Variety. The results seen in each of these categories are discussed bellow. ? Quantity Results: Of the five evaluation criteria, quantity was the only category which showed statistically significant results from every grader. This is likely due to the fact that quantity can be easily evaluated by counting the number of ideas. The numerically quantifiable nature of quantity makes it one of the more easily applicable evaluation criteria for research purposes. ? Quality Results: Quality is used to evaluate how well ideas address the given problem statement. However, some solutions primarily address problem symptoms whereas different solutions better address the problem’s root cause. The concept of identifying a problem’s root cause is likely a topic that had not been fully introduced to the participants. Some students’ ideas focused more on the symptoms of the problem, such as the damage caused by walking into objects, while other students created ideas aimed at addressing why people might be walking into objects in the first place. ? Feasibility Results: The perceived feasibility of an idea is affected by the participants ability to describe the required technology. Some students demonstrated more familiarity with the technologies proposed in their ideas. When a student has the experience to provide key technical detail it is easier for a grader to see the feasibility of the ideas. Fortunately, the graders’ experience in engineering was able to reduce this effect. The faculty graders were able to use their engineering experience to evaluate feasibility even when a student presented their ideas abstractly with less technical details. ? Novelty Results: To increase novelty students were encouraged to write down any ideas they thought of, even the seemingly crazy and less feasible ideas. The process of writing down wild ideas can often lead a designer to ideas that are both novel and feasible later in the brainstorming process. This instruction was observed in the results, with some students including impractical ideas in an effort to facilitate novel thinking. ? Variety Results: For this problem statement most students suggested using some variation of a cell phone app to solve the problem. This is not surprising considering that a cell phone was a central part of the problem statement. Some students did a better job of breaking away from this idea and explored other technology. It was clear that some students used the Elements of Design Ideation tool to systematically create ideas from each category. Some students used this systematic approach to create variations of the cell phone app idea while others used it to branch off into fundamentally different solutions. 7.4 Correlation of Evaluation Criteria The results of this study show a positive correlation between all factors used to evaluate creative output. Experimental Group 2 (multimodal) had a lower average score across all 5 of the evaluation criteria used in the grading rubric. In other words, all five factors send a consistent message about how “good” the ideation results were. In none of the statistically significant cases are conflicting results seen. For example, if the results had shown that Group 2’s quality was best while at the same time showing Group 1’s feasibility was best, one might conclude that the different versions of the tool lead to different areas of strength within the 28 results. However, since no such observations were seen, it could be argued that only one metric is needed for comparison of ideation results. Under this assumption, future ideation studies could benefit from a simplified evaluation method where only one metric was used. In such a case, an excellent choice would be to simply evaluate results on the quantity of ideas since this is the most easily quantifiable of all the evaluation criteria. In this way, future research could use quantity as a quick way to compare ideation results and gain a rough estimate of the overall behavior. 7.5 Sample Size and Participant Demographics It should be recognized that this study has a small sample size when compared to other large scale research projects on creativity. Due to time and resource constraints the scope of this study was limited to 32 participants divided into two experimental groups. Acknowledging this small sample size is important when interpreting the statistical results. The smaller a study’s sample size, the harder it is to draw generalized conclusions from the results. The statistics only compared two classes of students. It is entirely possible the all the high performing students happened to end up in the same class, thus artificially swaying the results. In order to get an idea about how evenly distributed the two experimental groups were, basic demographic information was collected. Figure 15 shows the distribution of age, GPA, engineering concentration, and gender across the two experimental groups. As it turned out, the two experimental groups contained a relatively even distribution of these demographic characteristics. The data reported in Figure 15 primarily serves to verify the random distribution of participants in the experimental groups. Without a more detailed investigation it cannot be determined whether a correlation exists between ideation results and any particular demographic metric. It should also be recognized that there are many other characteristics of the population which are not represented here, such as student motivation and effort. Participants were also asked if this was the first time they had used this style of engineering ideation tool. All but one participant answered “yes” that this was their first time using this type of tool. Figure 15: Participant Demographics 29 7.6 Discussion of Other Variables In this study, the independent variable was directly related to a characteristic of the Elements of Design Ideation Tool itself, namely the presence or absence of visuals. However, there are numerous other factors which influence ideation. While many of these factors were held constant during this experiment, there is still much value that can be gained from investigating their effect on ideation. The following section will list some of these factors and discuss their role in engineering ideation. ? Time - In this experiment 30 minutes were allowed for ideation. The amount of time students are given will greatly affect the range of ideas produced. In addition to duration, one must consider the number of sessions and amount of time between sessions. Two 30 minute sessions held on different days would likely produce different results than one 60 minute session. It is not unreasonable to hypothesize that too much concentrated time spent on ideation might start to yield diminishing returns. It would be interesting to identify the optimum schedule for performing ideation. ? Problem Statement - The type of problem and the way in which the problem statement is phrased can affect how designers understand the task. The medium used to present the problem should also be considered. In this experiment the problem was delivered as a written statement. However, ideation results would be affected if the problem were presented as a video or animation. It is likely that the Elements of Design Ideation Tool would have different results when used on different types of problems. In fact, an argument could be made that different ideation tools may be better suited to different types of problems. ? Instruction / Training Method - This experiment used prerecorded instruction to teach designers about the tool. Other instruction methods might lead designers to understand and use the tool differently. There are countless approaches to instructing content of this nature including interactive question and answer, group discussion, structured activates, etc. ? Experience - The amount of exposure to the content should be considered. Designers who are more familiar with different approaches to conceptual design would likely use an ideation tool differently. In addition, it is likely that results would change if participants were familiar with the Elements of Design Ideation Tool. An investigation could be done to see how experience level affects a designer’s use of ideation tools. ? Team Ideation - Ideation in a group setting is a significantly different environment. As such, ideation tools would likely perform differently when used by a team verses an individual. When it comes to team ideation, there are many factors to consider which can influence the group’s dynamic and productivity. Some tools which prove useful for individuals may be less valuable for a group, and the inverse case as well. ? Ideation Setting - The physical setting and environment where ideation takes place can influence a designer’s output. In this study, students conducted their brainstorming session in a traditional classroom setting. It would be interesting to see how ideation results are affected if the assignment was set as a take home activity. Doing this experiment as a take home assignment would limit the researcher’s control over the experiment. Nevertheless, it would be interesting to see whether the flexibility and comfort of doing an ideation session at home improved results or simply introduced more distractions. It should be noted that the particular configuration of variables used to set up this experiment led to results where the verbal tool performed better. However, this does not necessarily mean 30 that the multimodal tool would not prove more effective under a different combination of factors. All in all, there are many factors to be considered when studying how to improve engineering ideation. Researchers should consider how all these factors work together to create the best overall ideation environment. 7.7 Discussion of Further Research The experimentally structured nature of this research introduced certain constraints pertaining to the study’s methods and approach. For example, in order to control as many variables as possible, the instruction given to students about the Elements of Design Ideation Tool was delivered using a prerecorded video in each class. It could be beneficial to repeat this experiment with a more interactive instruction method where students are given the opportunity to ask questions, practice collaboratively with peers, and receive real time feedback. Additionally, instructors might encourage students to practice creating their own examples for each category of the tool. It would also be instructive to do a longer duration study interviewing students throughout the learning experience to see how their impressions of the tool change with more time and practice, much like Watkins’ approach [8]. Finally, an interesting modification to the Element of Design Tool would be to investigate the effects of using colored icons, like those shown in the TRIZ Ideation Tool, Figure 4 [13]. It would be fascinating to see if color influences the way designers engage with the Elements of Design Ideation Tool. 31 8. CONCLUSION This study investigated the hypothesis that a multimodal engineering ideation tool would lead to better creative output when compared to a strictly verbal tool under similar conditions. However, the results did not show evidence to support this claim. In fact, the results pointed rather uniformly to the opposite conclusion. To understand these results a discussion was made exploring why the addition of icons may not have been beneficial when employed in this way. Furthermore, it was recognized that an experiment of this nature has numerous variables which come together to build the overall design context. Conceptual design and ideation are open ended. It is not as if there is just one way to approach design. With this understanding it is reasonable to assert that in a different design setting the multimodal tool may have performed better, even though it did not in this particular scenario. It was also observed that all five evaluation criteria used in the grading rubric were positively correlated. This correlation suggests that the differences in the two versions of the tool did not affect any one element of the results more than it affected another. In the end, all five criteria delivered the same message about the overall behavior of the results from the two experimental groups. This suggests that further works, depending on the intent of the research, could simply use one evaluation metric for comparing the performance of creative output. Perhaps one of the strongest lessons from this experiment is the impact of instruction on ideation results. The way a designer is taught has a significant influence on the set of ideation results they produce. More specifically, it was seen that the techniques that an instructor focuses on most will be reflected in their students’ work. In this way, instructional design holds the potential to influence design ideation. 32 References [1] J. Hirtz, R. B. Stone, D. A. McAdams, S. Szykman, and K. L. Wood, “A functional basis for engineering design: Reconciling and evolving previous efforts,” Research in Engineering Design, vol. 13, pp. 65-82, 2002. Retrieved from: https://link.springer.com/article/10.1007/s00163-001-0008-3 [2] K. Goucher-Lambert, J. T. Gyory, K. Kotovsky, and J. Cagan, “Adaptive Inspirational Design Stimuli: Using Design Output to Computationally Search for Stimuli That Impact Concept Generation,” Journal of Mechanical Design, vol. 142, September 2020. Retrieved from: https://doi.org/10.1115/1.4046077 [3] C. A. Toh and S. R. Miller, “The Impact of Example Modality and Physical Interactions on Design Creativity,” Journal of Mechanical Design, vol. 136, September 2014. Retrieved from: https://doi.org/10.1115/1.4027639 [4] E. B. Scamper, “Scamper: Let your imagination run wild,” Prufrock Press, Waco, TX, 2008. [5] T. Ritchey, “General Morphological Analysis: A general method for non-quantified modelling,” Swedish Morphological Society, 2002. Retrieved from: http://www.swemorph.com/pdf/gma.pdf [6] R. Felder and L. Silverman, “Learning and teaching styles in engineering education,” Engineering Education, vol. 78, no. 7, pp. 674–681, 1988. Retrieved from: https://www.researchgate.net/publication/257431200 [7] CAST, “The UDL Guidelines,” cast.org. Retrieved from: https://udlguidelines.cast.org/ [8] R. D. Watkins, “Sequential rhetoric: Teaching comic as visual rhetoric,” Iowa State University: Graduate Theses and Dissertations, 2014. Retrieved from: https://lib.dr.iastate.edu/etd/13940/?utm_source=lib.dr.iastate.edu%2Fetd%2F13940& utm_medium=PDF&utm_campaign=PDFCoverPages [9] J. T. Holden, “Visuals In E-Learning: What We’ve Learned from Cognitive Science,” 2015. Retrieved from: https://enliten.net/dev/wp-content/uploads/2018/06/VISUALS-IN-e-LEARNING-Wh at-Weve-Learned-from-Cognitive-Science-Holden-15-10-12.pdf [10] R. E. Mayer and R. Moreno, “Nine Ways to Reduce Cognitive Load in Multimedia Learning,” Educational Psychologist, vol. 38, no. 1, pp. 43–52, 2003. Retrieved from: http://www.uky.edu/~gmswan3/544/9_ways_to_reduce_CL.pdf [11] K. Haramundanis, “Why Icons Cannot Stand Alone,” ACM SIGDOC Asterisk Journal of Computer Documentation, vol. 20, no. 2, pp. 1–8, 1996. Retrieved from: https://dl.acm.org/doi/pdf/10.1145/381815.381819 [12] D. G. Morrow, V. O. Leirer, and J. M. Andrassy, “Using icons to convey medication schedule information,” Applied Ergonomics, vol. 27, no. 4, pp. 267–275, 1996. Retrieved from: https://www.sciencedirect.com/science/article/pii/0003687096000117 [13] R. Cordero, “TRIZ – The theory of solving inventive problems,” blog post, 10 July, 2018. Retrieved from: http://www.rodrigocordero.org/en/triz-the-theory-of-solving-inventive-problems/ [14] R. H. Sturges JR, K. O’Shaughnessy, and R. G. Reed, “A Systematic Approach to Conceptual Design,” Concurrent Engineering, vol. 1, no. 2, pp. 93–105, 1993. Retrieved from: https://doi.org/10.1177/1063293X9300100202 [15] R. L. Nagel, M. R. Bohm, J. S. Linsey, and M. K. Riggs, “Improving Students’ Functional Modeling Skills: A Modeling Approach and a Scoring Rubric,” Journal of Mechanical Design, vol. 137, May 2015. Retrieved from: https://doi.org/10.1115/1.4029585 [16] H. Pashler, M. McDaniel, D. Rohrer, and R. Bjork, “Learning Styles: Concepts and Evidence,” Psychological Science in the Public Interest, vol. 9, no. 3, pp. 105-119, December 2008. Retrieved from: https://www.jstor.org/stable/20697325 [17] P. M. Newton and M. Miah, “Evidence-Based Higher Education - Is the Learning Styles Myth Important?,” Frontiers in Psychology, vol. 8, March 2017. Retrieved from: https://doi.org/10.3389/fpsyg.2017.00444 [18] R. M. Felder, “Opinion: Uses, Misuses, and Validity of Learning Styles,” Advances in Engineering Education, vol. 8, no. 1, May 2020. Retrieved from: https://www.engr.ncsu.edu/wp-content/uploads/drive/1tKCP5oEAV5VV4Yb97j-IG_ geuBxCQqB6/2020-AEE%20Learning%20Styles%20Opinion%20Piece.pdf [19] R. Roy and J. P. Warren, “Card-based design tools: a review and analysis of 155 card decks for designers and designing,” Design Studies, vol. 63, pp. 125-154, July 2019. Retrieved from: https://doi.org/10.1016/j.destud.2019.04.002 [20] J. Shah, N. Vargas-Hernandez, and S. M. Smith, “Metrics for measuring ideation effectiveness,” Design Studies, vol. 24, no. 2, pp. 111–134, 2003. Retrieved from: https://www.sciencedirect.com/science/article/pii/S0142694X02000340 [21] The Noun Project, “Icons and Photos For Everything.” Accessed on March 12, 2021, https://thenounproject.com 34 Appendix A: Elements of Design Ideation Tool Icons in the Elements of Design Ideation Tool are adapted from a collection of varying artists from [21]. Figure A.1: Group 2 Ideation Tool with both Text and Visual Elements (Multimodal) Primary Secondary Tertiary Examples Human Operator Consumer Manager Investors Competitors Co-Workers Gas Blimp Steam Engine Steam Iron Neon Light Scuba Freon Liquid Mercury Oil/Grease Paint Blood Condensation Gasoline Object Insulator / Conductor Ceramic Fabric Glass Metal Plastic Solid Particulate Friction Enhancer Pollen Sand Powder Sugar Fertilizer Material Composite Kevlar Reinforced Concrete Carbon Fiber Plywood Fiberglass Honeycomb Structure Plasma Plasma Ball Rocket Propulsion Sun Lightning Auroras Static Electricity Solid-Liquid Toothpaste Glue/Gel Dental Filling Mold/Casting Permeable Membrane Salt Water Mixtures Liquid-Gas Carbonation Aerosols Humidifier Whipped Cream Foam Fog/Clouds Solid-Gas Insulated Window Cushion Filter Sponge Smoke/Ash Styrofoam A2 Primary Secondary Tertiary Examples Auditory Alarm Speaker Bell Buzzer Audio Recording Tuning Fork Olfactory Scented Natural Gas Cooking Smoke Air Quality Perfume Tracking Dog Status Tactile Vibration Temperature Brail Mouse Click Weight Texture Signal Taste Sweet Bitter Sour Salty Spicy Dry Visual Visual Gauge Displacement Gauge Indicator Light Microscope Video Recording Traffic Light Analog Analog Clock Knob Signal Filter Joy Stick Sliding Caliper Analog Signal Control Discrete Digital Clock Switch Key Pad Arrow Keys Telegraph Binary Code A2 Primary Secondary Tertiary Examples Human Hole Punch Blade Brush Crank Handle Jack Acoustic Acoustic Insulator Stethoscope Echolocation / Radar Sonic Boom Pipe Organ Vocal Cords Biological Photosynthesis Digestive System Horse Carriage Stem Cells Vaccine Algae Biomass Chemical Catalytic Converter Air Bags Combustion Engine Glow Stick Corrosive Hot / Cold Pack Electrical Battery Capacitor AC Current Diode Electric Motor Resistor Electromagnetic Radio Waves Laser Cutter Light Prism Microwave X-Rays Solar Panel Energy Hydraulic Pressure Washer Water Wheel Valve Nozzle Hydraulic Pump Hydraulic Excavator Magnetic MRI Armature Electromagnet Compass Credit Card Attract / Repel Rotational Lever Centrifuge Agitator Bearing Mechanical Transformer Cam Mechanical Translational Telescoping Belt Pistons Spring Link Pulley Fasteners Clip Latch Key Rivets Bracket Hinge Pneumatic Tier Pneumatic Tools Wind Mill Air Foil Fan Air Compressor Radioactive/Nuclear Power Plant Radiation Therapy Nuclear Explosion Rocket Propulsion Food Treatment Smoke Detectors Thermal Heating Element Burner Thermal Wire Solder Heat Exchanger Inductor A2 Figure A.2: Group 1 Ideation Tool with only Text Elements (Verbal) Primary Secondary Tertiary Examples Human Operator Consumer Manager Investors Competitors Co-Workers Gas Blimp Steam Engine Steam Iron Neon Light Scuba Freon Liquid Mercury Oil/Grease Paint Blood Condensation Gasoline Object Insulator / Conductor Ceramic Fabric Glass Metal Plastic Solid Particulate Friction Enhancer Pollen Sand Powder Sugar Fertilizer Material Composite Kevlar Reinforced Concrete Carbon Fiber Plywood Fiberglass Honeycomb Structure Plasma Plasma Ball Rocket Propulsion Sun Lightning Auroras Static Electricity Solid-Liquid Toothpaste Glue/Gel Dental Filling Mold/Casting Permeable Membrane Salt Water Mixtures Liquid-Gas Carbonation Aerosols Humidifier Whipped Cream Foam Fog/Clouds Solid-Gas Insulated Window Cushion Filter Sponge Smoke/Ash Styrofoam A6 Primary Secondary Tertiary Examples Auditory Alarm Speaker Bell Buzzer Audio Recording Tuning Fork Olfactory Scented Natural Gas Cooking Smoke Air Quality Perfume Tracking Dog Status Tactile Vibration Temperature Brail Mouse Click Weight Texture Signal Taste Sweet Bitter Sour Salty Spicy Dry Visual Visual Gauge Displacement Gauge Indicator Light Microscope Video Recording Traffic Light Analog Analog Clock Knob Signal Filter Joy Stick Sliding Caliper Analog Signal Control Discrete Digital Clock Switch Key Pad Arrow Keys Telegraph Binary Code A6 Primary Secondary Tertiary Examples Human Hole Punch Blade Brush Crank Handle Jack Acoustic Acoustic Insulator Stethoscope Echolocation / Radar Sonic Boom Pipe Organ Vocal Cords Biological Photosynthesis Digestive System Horse Carriage Stem Cells Vaccine Algae Biomass Chemical Catalytic Converter Air Bags Combustion Engine Glow Stick Corrosive Hot / Cold Pack Electrical Battery Capacitor AC Current Diode Electric Motor Resistor Electromagnetic Radio Waves Laser Cutter Light Prism Microwave X-Rays Solar Panel Hydraulic Pressure Washer Water Wheel Valve Nozzle Hydraulic Pump Hydraulic Excavator Energy Magnetic MRI Armature Electromagnet Compass Credit Card Attract / Repel Rotational Lever Centrifuge Agitator Bearing Mechanical Transformer Cam Mechanical Translational Telescoping Belt Pistons Spring Link Pulley Fasteners Clip Latch Key Rivets Bracket Hinge Pneumatic Tier Pneumatic Tools Wind Mill Air Foil Fan Air Compressor Radioactive/Nuclear Power Plant Radiation Therapy Nuclear Explosion Rocket Propulsion Food Treatment Smoke Detectors Thermal Heating Element Burner Thermal Wire Solder Heat Exchanger Inductor A6 Appendix B: Transcript of Experiment Instructions Voluntary Consent Notice: You are being invited to participate in a research study titled “Increasing Creative Output by Visually Enhancing Engineering Design Tools” being conducted by David Harr, a graduate student at East Carolina University in the Department of Engineering. The goal is to survey 60 engineering sophomore students enrolled in ENGR-2000. The study will take approximately 50 minutes to complete. It is hoped that this information will improve the way engineering design tools are used. The data collected during the experiment will be used in the accompanying research. Your responses will be kept confidential, such that no data will be released or used with your identification attached. Your participation in the research is voluntary. You may choose not to answer any or all questions, and you may stop at any time. We will not be able to pay you for the time you volunteer while being in this study. There is no penalty for not taking part in this research study. Results from participants under age 18 are not eligible to be used for this research. Feel free to contact David Harr (Harrd18@students.ecu.edu) for any research related questions or the University & Medical Center Institutional Review Board (UMCIRB) at 252-744-2914 for questions about your rights as a research participant. Engineering Ideation Research Study Instructions: Using the Elements of Design Ideation Tool you were provided, generate a list of alternative solutions to the following problem statement. When listing ideas, it is good to list the name of the idea and provide a short one or two sentence description so the graders will fully understand the concept of your idea. For example… Name of First Idea – One or two sentence description of the concept. Name of Second Idea – One or two sentence description of the concept. … Problem Statement: Many individuals operate a cell phone while walking. The problem is, they walk into obstacles and people since they are not looking where they are walking. Use the space bellow to create your list of ideas: 42 Post Research Participation Demographic Survey Please Note: Providing the following information is voluntary. This information is used to give the researchers a better idea of who participated in the study. 1. Age: ___________ 2. Gender: ______________ 3. GPA: ________ 4. Ethnicity: __________________________________ 5. Engineering Concentration: (Please Circle One) Biomedical Environmental Bioprocess Industrial and Systems Electrical Undecided Mechanical Other: ______________________ 6. Is this the first time you have used this style of engineering ideation tool? Yes / No 7. Participation in this research study is voluntary. Please confirm that you consent to having your results used to further this research. a. I would like my results to be used to benefit this research b. I do not want my results to be used in this research 43