Dec 3rd Begining of day.

Dr Westberry  came in and discussed changes at the SIU Campus.

We completed a survey at the following link http://www.surveymonkey.com/s/HT7SSHC

The class is coming to an end and Dr. Ackerman started to review and discuss the syllabus and look at bring the course to a close.

Next we went into reviewing our webquests. The intent was to take the feedback of our peers and with a partner review our webquests and look at what we would change or update.

My webquest was on Fantasy Football. My partner for this exercise was Angela Miller and her webquest was on How to make Banana Pudding. We reviewed each others feedback; Angela's and Bill's. From my peers feedback I've went back to my wiki individual page and identified themes throughout my webquest. After looking at the feedback from my peers we decided that it was evident that I relide on videos to complete my task. I should have supported them with some text. My intent was to hit multiple levels of learning. I knew that some students learning better by listening and figured I could blend that style of learning with reading text. My review should have included grammar and puncuation break downs. The organization of the webquest links on the left hand side should have been reorganized to have the post test prior to the conclusion. The post and pre-tests should have been more difficult and been worded differently. I also should have restated the objective.

I learned alot from this process.

Next we went into the Eval levels scenarios. We dicussed the different levels of the frameshop scenario.

Next we went through scenarios and evaluated using the below formulas;

RETURN ON INVESTMENT FORMULAS

 

Benefit/Cost Ratio (BCR) and  ROI

Benefit/Cost Ratio    (Uses total benefits and costs)

BCR = Program Benefits

              Program Costs

 

ROI  (Costs are subtracted from total benefits for net benefits ¸ costs)

ROI% = Net Program Benefits     x 100

               Program Costs

  

=============================================================

For example, a communication skills program produced benefits of $321,600

 with a cost of $38,233.  Thus, the benefit/cost ratio would be:

BCR=  $321,600    = 8.41 or (8.41:1)

               $38,233

 

For every $1 invested, $8.41 is returned in benefits.

 

=============================================================

In this example, net benefits are $321,600 - $38,233 = $283,367.  Thus, the ROI

 

ROI = $283,367  X 100 = 741%

            $38,233

 

For every $1 invested, $7.41 is returned in net benefits.

 

=============================================================

Note that benefits are usually expressed as annual benefits for a complete year after the program is implemented.  While the benefits may continue after the first year, the impact usually diminishes and is omitted from calculations.  The values for ROI may be quite large, in the range of 25% to 500% in many organizations.

The picture illistrates work completed in the classroom.

Next task was to develop 1 Quiz questions per team, multiple choice, With a High Level (Bloom) not look it up with a short scenario. This is our response.

Team Amanda, Bill, Angela & Ryan
Directions: Read the short scenario below, and then circle the letter corresponding with the correct answer.
Scenario: Katie owns a small newspaper company. Due to recent road construction, she had to develop a new delivery route to ensure timely delivery of her paper. What level of evaluation did Katie use?
A) Apply
B) Analyze
C) Evaluate
D) Create
Justification: Once Katie realized there was road construction, she had to analyze all possible routes to get to her customers. She then had to evaluate each possible route to determine which is the quickest and most cost effective. Last, she had to create the new delivery route.

Manny, Nick, Steve Final Test Question:Given the following, place an X next on the line to the left of the letter for the correct Kirkpatrick Level of Evaluation from the list of possible answers that best describes the scenario.

Johnny joined the Air Force as an aircraft mechanic and went to the aircraft maintenance technical school in Texas. After completing his school, he went to his permanent duty station. After six months of working at his assigned aircraft maintenance shop, his supervisor received a survey from the technical school in Texas. This survey asked specific questions to the effectiveness of Airman Johnny's skills as an aircraft mechanic within the work section.

a) Level 3b) Level 2
c) Level 4d) Level 1

CA: a
Rationale: (a) is the correct answer as the survey is asking how the transfer of information that Johnny learned in school is impacting the work section.

Team Member: David, Javed, and Sade

Instructions: 

Read the scenario below and select the correct answer for the following multiple choice question. Indicate the correct answer(s) by an X in the space provided. Select all that apply.

Scenario:

An employee not following proper fall protection procedures while working at heights receives a write-up and dismissal because this behavior can RESULT in a serious injury or death. In the event of injury, the Air Force incurs losses in productivity, medical expenses, and possible lawsuits—not to mention the effect an injury can have on the employee’s family or morale on the job site.

Q1: If a serious injury or death occurred, what could be the ending result(s)?

___ Legal actions or lawsuits

___ Lower employee morale

___ Decrease employee retention

___ Higher employee morale

Group: Duong, Al, Rantz

High Level Question

The Philadelphia Zoo was tasked with an initiative to lower their overall water usage at the zoo. After extensive brainstorming on ways of which they can reduce their water usage the board was presented by senior leadership with four choices. Given the formula for calculating return on investment (ROI). Which of these four choices presents the best ROI percentage for the zoo? Choose only one answer and circle the corresponding letter.

ROI=Benefits/Cost x 100

1. 1.5 Million dollars saved/1.0 million in total cost
2. 3.9 Million dollars saved/2.5 million in total cost
3. 2.5 Million dollars saved/1.0 million in total cost
4. 5.7 Million dollars saved/5.0 million in total cost

answer : C

Lastly we did a 90sec reflection. Everyone learned from the course.

17-18 Nov E-learning weekend.

Since last I blogged we've been knocking out our webquest assignments. My webquest was about Fantasy Football. The overarching feeling that I had in regard to this assignment was a feeling of overwelming. I thought that this assignment was great. After looking at my classmates webquests, I truly realize that I could have done a better job. Morale of the story is that the work I put in was good but it could have been better.

How does training evaluation take place in your workplace?
When a new instructor arrives in our unit they are entered into what is termed "Phase One" training. During this period the intent is for each new member to accomlish all the tasks necessary to become instuctors. New instructors attended Acidemic Instructor Course in addition to initial unit level training. These individuals learn the basics of researching lesson plans and developing power points to support them. After completion of this phase one process, new instuctors are released from training and enter "Phase Two" training. This training cycle begins their practical On The Job Training (OJT) . This training is evaluated during Stan Eval and instructor evals. The evals are conducted by while an instructor is observed. This evaluates the instructors comprehension and retention of training they recieved during both phases of training.

How might you use evaluation tools outside of the workplace?
I use evalution tools with my seven year old and two year old. I also evaluate my ability to listen and respond to my wife.

What techniques might be used to evaluate performance of a team (as whole) and individual members? Include a unique, relevant link.
During the courses I teach at the USAF EC we teach several tasks. Some of these are called battle drills. These battle drills are taught to squads or groups of 9-13 people. These personnel are evaluated on their performance of the steps for each of the battle drills they're taught. Creating task performance checklists allow for students to evaluated on collective and individual tasks. Evaluating learner performance in this way allows for a practical evaluation of all members of team.

Create a brief scenario that illustrates Level 3 (Kirkpatrick) in your environment or a provide a hypothetical one.
Level 3 evaluations measure whether there was a transfer of learning to the actual work setting. This level of evaluation affects learners, managers, project stakeholders, subject matter experts (SMEs), and line supervisors. It also relates to mentoring programs, annual performance reports and development planning. After the courses in the USAF EC our students and their suppervisors recieve a Level Three feedback sheet. This feedback sheet asks both of these customers weither our training met the deployers needs.
 

Dr. Koller Video
Daphne Koller is enticing top universities to put their most intriguing courses online for free -- not just as a service, but as a way to research how people learn. With Coursera (cofounded by Andrew Ng), each keystroke, quiz, peer-to-peer discussion and self-graded assignment builds an unprecedented pool of data on how knowledge is processed.
With Coursera, Daphne Koller and co-founder Andrew Ng are bringing courses from top colleges online, free, for anyone who wants to take them.

Many of these courses are interesting. I would enjoy attending many of these courses.

This shows the potential of future e-learning. Much of this could be applied to the USAF EC. Our courses are more performance based, but many of the tasks taught in our courses could be completed prior to arrival through technologies like these.

Reflections on Day One of our second weekend (3-4 Nov). These are my stream of conciousness notes for the day.

Discussion began with our ISD exercise and in particular Question #4.

Learner Performance is the focus of each stage  of the model.

Formatively Evaluate Each Design Stage

ADDIE
Revise as Required: The five phases are ongoing activities that continue throughout the life of a training program. After building a training program, the other phases do not end once the training program is implemented. The five phases work like a loop. They are continually repeated on a regular basis to see if further improvements can be made.

Always Track Performance for process improvement.

LL Bean was the bench mark for how inventory control is impotant to your bottom line.

Proctor and gamble discussed inventory control with WALMART. When the inventory of a product drops to a certain level a shipment will be sent to replenish that inventory.

What is your tolerance for any variance in training effectiveness.

Looking at ROI according to constant evaluation. Use spikes or dips in metrics as an opportunity to evaluate your program.

Discussion turned to the flipped classroom. Some of the pros and cons of this learning technique was discussed.
http://www.flippedclassroom.com/

-Rapid Prototyping. 50-60% is spent on analisys and design.
-Make prudent estimates with data.
Story Boards being used to present information.

Constant feedback from data collection allows course developers to focus on propper implimentation of training.

1hour of delivery can take 100-300 hours of development.

MOOC A massive open online course (MOOC) is a type of online course aimed at large-scale participation and open access via the web. MOOCs are a recent development in the area of distance education, and a progression of the kind of open education ideals suggested by open educational resources.

All this is based on collaberative learning.

Minimize test question avoidance. When developing questions and testing instruments, make sure that your questions meet your intent.

 Millers Magic Number.

Primacy Recency Effect

 

 

When you have the students’ focus, teach the new information. Don’t let prime-time get contaminated with wrong information.

Looking at the Seriel position effect.

Then we turned to the do they match exercise.
 
Day two dicussion was focused on our Webquest projects and question development.

The above question was posted and each member was directed to goto the individual page menu.

Student responses were reviewed and discussion focused the good bad good of each students questions.



HOTS and LOTS

The Above diagram was discussed. We looked at the importance of performance evaluation that all your performance can't be measured without having a control group to measure against.

Also discussion was turned to picking what 2 out of the three choices are the most important.

Pick2
Time
Quality
Cost.

Review of the E- Learning Weekend was conducted. First goto Assignments.

Webquest review directions:

-Examine each webquest (see wiki discussion for links) -Complete each wq QUIZ by using a name we recognize (if you want credit) -For each wq review, post the following in review wiki discussions designated for each review and in the actual wq if you are logged into zunal (optional). -Your recognizable name (often not wiki user name) -Name of webquest you are reviewing -2 unique observations about wq --refer to this wq rubric http://webquest.sdsu.edu/webquestrubric.html -2 observations about the quiz (refer to our guidelines) -2 recommendations/considerations for future revision/enhancement of this wq As professionals, please state your reviews in a unique manner so they don’t appear as if you just made copies without careful consideration. The reviews should demonstrate that you examined the individual webquests. Lots of creative approaches and application of 463 in terms of assessment and evaluation as well as this formative evaluation technique (review process). Congrats – AA



This post is for my out of class assignment from 21 October. my answers for them are below

7. Because adult learners are task-centered, adult training should be task-centered. Consequently, adult learning should be based on measurable, task-centered learning outcomes. Inherent to good instructional design is the definition of clear learning outcomes. Dr. Robert Mager, a well-known figure in training design, defines learning outcomes (which he calls “learning objectives”) in this way:

Robert Mager on Learning Objectives:

“[Learning] Objectives are a little like blueprints. They provide the guides that will guarantee that you are teaching what needs to be taught…Statements describing intended instructional outcomes are called objectives because their accomplishment can be measured. Goals are broad (fuzzy) statements of intent; objectives are measurable statements of intent. In plain language, if an outcome statement isn’t precise enough to measure whether the outcome has been achieved, it isn’t an objective.”

“An objective describes student performance. It doesn’t say anything about what the instructor will do or try to accomplish. It doesn’t describe course content or the textbook.”

“An objective is about end rather than means. It describes a product of instruction rather than the process of instruction. It describes what students will be able to do when the are competent, rather than describing how they will be made competent.”

Robert F. Mager, Making Instruction Work, pages 73-75

8. It is often difficult to take the time necessary to follow the ISD process. Trainer get wrapped up in the tasks rather than how they fit together. Even though as I found online; “The ISD model is a tool for solving many types of performance problems” often times instructors get caught up in the minutia of the tasks.

9. The ISD process, by definition is; “a systematic process that is employed to develop education and training programs in a consistent and reliable fashion” (Reiser & Dempsey, 2007). In my opinion the ISD process is like creating football plays. The teacher/Instructor is the coach and the students are the players. The coaches take the task and break it into parts. First they define the objective, “winning the game” and they backwards plan from there. During this planning they develop the players and coach the players to achieve certain objectives, “block and tackle”. After they train the players to perform the lower level tasks they create the plays that eventually become the playbook. The formative checks on learning are the practices. Then eventually, on game day, the summative evaluation occurs. This is how I understand the ISD the Model.  

Reflections on Day ONE and TWO (20-21 Oct 12)

When using this BLOG be sure that you keep an eye out for hyperlinks to additional information.

Reflections on Day ONE:

Day one was a whirlwind. After stopping in to work, I showed up to class late. This was a big mistake. After realizing that time management is my problem, not the instructors, we completed a visual learning exercise.  The intent of this visual learning exercise was to have students to visually depict who they are and descride how they prefer to learn in pictures. Students were paired up and instructed to stand-up and talk about their partners. From this point discussion turned to the course syllabus. During the discussion of the syllabus, there were many references to social media and its use in the classroom. The instructor spoke about the impact of technology and the implication of its use.

Discussion turned to the use of evaluation as a check on the trainer. One of the points that was dicussed was the difference between Education, Presentation/Lecture, and Training.
Education was the stated to be increasing knowledge.
Presentation on the otherhand was discussed and stated to be "Lecturing" not teaching or training. Lecturing was stated to be when a trainer or instructor stands up and puts students in the "recieve" mode vesrus actually having them interact.

Training was discussed to be"deliberate set of activities to improve job performance. This includes practice and feedback on the skills taught."

The importance of Evaluation was discussed and the gap between performance and evaluation.

Gap Analysis was dipected using the following diagram;

This was summed up as the students in the course are performance consultants.



Reflections on Day TWO:

Day TWO started with completing a quiz. As with all quizes in this course, this was an open book quiz with the syllabus. After completion of the quiz, work turned to our blogs to ensure we could access them.

Discussion turned to back-channeling. The talk also turned to Multi-Tasking and research about adolescents being "digital natives".

Formative evaluation.

Five Levels of Evaluation.

 



1. Reaction: (Is about the "Smiles" test)

2. Learning: (Pre/Post Tests)

3. Transfer: (Back to the Job)

4. Business Impact: (Numbers)

5. Return on Investment: ($) Cost

Understanding metrics and how there are demographic impacts to the success of training.

 

 

 

Discussion was focused on how there is a substantial difference between Levels Four and Five. The business impact can have a negative impact on short term decision making. Metrics should be looked at with regard to longetivty of the decision making.



Classroom exercise was too:

1. Locate guideline on the web: my guidelines are hyperlinked to the right; (search for writing items).

2. With a partner critique three test items on the handout.

In conjunction with the Guidelines for writing test items I found on the internet we indentified several items wrong with the test.

Item #1 had several errors; was the the fact that the test item had spelling and syntax errors. Also the test item had negative ideas.

Item#2 had several errors as well. Spelling errors. It also failed to follow some of the guidlines for mutliple choice test items. An example would be: Multiple-Choice Test Items #5; Avoid “None of the above,” “Some of the above,” “All of the above,” phrases which usually scream out that they are the correct answer.

Item#3 had several issues as well, two many blanks, gramatical and syntax errors. Also: Completion-Test Items #6; Limit the number of blanks to one or two per item. Statements with too many blanks waste time as students figure out what is being asked.

The next classroom exercise was: with a partner create 1 multiple choice item, evaluation related. My partner and I completed this with the below question:

 

 

This question was intended to evaluate the students understanding of the requirements for blog creation. What we forgot was how to properly complete the test. Direction such as; select all that apply according to the course syllabus. For our question we could have also gotten the desired outcome using a checklist style.

Remember that one word has an impact.

Each group had questions that had suttle flaws. The focus of the instructor was to ensure precision and brevity of questions. When conducting a formative review we should be sensitive to the needs of students but don't succumb to the pressures of meeting every potential students needs.

The class concluded with a review of the exercise. Each groups questions were discussed. Students discussed stories and pesonal experience.

 The out of class assignment was given to students and discussed;
 

 

Item number 4 was stated to be hard to find or very rare for s student to answer.

Too answer the questions the instructor will establish a WIKI location for students to post their answers.

An internet search was conducted on google and this site was found; ISD INFO

Brief Description of the ISD Process

1. **Analyze**

• Determine business outcome or linkage.
• Analyze system (department, job, etc.) to gain an understanding of it.
• Compile a task inventory of all tasks associated with each job (if needed).
• Select tasks that people need to learn to become performers (needs analysis).
• Build performance measures for the tasks to be learned.
• Choose instructional setting for the tasks to be learned, e.g. classroom, elearning, on-the-job, self study, blended, etc.
• Estimate cost and compare to benefits gained.

1. **Design**

• Develop the learning objectives, to include both terminal and enabling objectives.
• Identify and list the learning steps required to perform the task.
• Develop performance tests to show mastery of the tasks.
• List the entry behaviors that the learner must demonstrate prior to entering the learning program.
• Sequence and structure the learning objectives.

1. **Develop**

• List activities that will help the students learn the task.
• Select the delivery methods (media).
• Review existing material so that you do not reinvent the wheel.
• Develop the instructional courseware.
• Synthesize the courseware into a viable learning program.
• Validate the instruction to ensure it accomplishes all goals and objectives.

1. **Implement**

• Create a management plan for conducting the training.
• Conduct the training.

1. **Evaluate**

• Review and evaluate each phase (analyze, design, develop, implement) to ensure it is accomplishing what it is supposed to.
• Perform external evaluations, e.g. observe that the tasks that were trained can actually be performed by the learners in their working environment.

Revise training system to make it better and to meet future challenges
 

The First Post Ever

This is my first time posting. Lets see how this goes.