Mathematics 209: Finite Mathematics

Study Guide

Unit 7: Properties of Markov Chains

In this unit, you will learn another application of matrices. You will analyse a stochastic process, which is a sequence of trials that satisfy certain conditions. These sets of trials are called Markov chains.

Objectives

When you have completed this unit, you should be able to

  1. construct the transition matrix for a Markov chain, and compute successive state matrices.
  2. identify regular and absorbing Markov chains.
  3. determine the long-run behaviour of a regular Markov chain.
  4. determine the long-run behaviour of an absorbing Markov chain.

Properties of Markov Chains

Indications

  1. Read Section 9-1, Properties of Markov Chains, on pages 353–360 of the textbook.

    2. Note: The probability tree mentioned on page 354 refers to Chapter 8 of the original textbook from which this customized version was derived. The discussion referred to is limited to using the product of matrices to obtain the state of matrices. It is not necessary to your understanding in this course.

  2. Do odd-numbered exercises 1–51 on pages 360–362 of the textbook. If you have difficulty, consult your tutor to discuss the problem.
  3. Solve odd-numbered problems 57–67 on pages 363–364 of the textbook. If you have difficulty, consult your tutor to discuss the problem.
  4. Answer the questions listed below, and then compare your answers with those given in the Answers to Study Guide Questions.

Questions

  1. In a transition matrix, what is the meaning of the diagonal entries a11, a22?
  2. What is an absorbing state?
  3. If a transition matrix has a 1 as one of the entries on the main diagonal, what do we know about its state?
  4. Why is a transition matrix always square?

Regular Markov Chains

Indications

  1. Read Section 9-2, Regular Markov Chains, on pages 364–370 of the textbook.
  2. Do odd-numbered exercises 1–37 on pages 370–372 of the textbook. If you have difficulty, consult your tutor to discuss the problem.
  3. Solve odd-numbered problems 39–53 on pages 372–374 of the textbook. If you have difficulty, consult your tutor to discuss the problem.
  4. Answer the questions listed below, and then compare your answers with those given in the Answers to Study Guide Questions.

Questions

  1. What does a stationary matrix indicate about a problem?
  2. What ensures the existence of a stationary matrix in a Markov chain?
  3. Can a transition matrix have two different stationary matrices?

Absorbing Markov Chains

Indications

  1. Read Section 9-3, Absorbing Markov Chains, on pages 374–385 of the textbook.
  2. Do odd-numbered exercises 1–47 on pages 385–387 of the textbook. If you have difficulty, consult your tutor to discuss the problem.
  3. Solve odd-numbered problems 55–59 on page 388 of the textbook. If you have difficulty, consult your tutor to discuss the problem.
  4. Answer the questions listed below, and then compare your answers with those given in the Answers to Study Guide Questions.

Questions

  1. How do we recognize a possible absorbing Markov chain?
  2. Is it true that a Markov chain with two states, one nonabsorbing and one absorbing, is always an absorbing chain?
  3. What are the components of a transition matrix in an absorbing Markov chain?
  4. What is the meaning of the limiting transition matrix of a Markov chain?

Finishing This Unit

  1. Review the objectives of this unit and make sure you are able to meet each of them.
  2. Study the section of the Chapter 9 Review titled Important Terms, Symbols and Concepts on pages 389–390 of the textbook.
  3. If there is a concept, definition, example or exercise that is not yet clear to you, go back and re-read it. Contact your tutor if you need help.
  4. You might want to do odd-numbered exercises 1–37 from the Review Exercise section on pages 390–392. The questions on the practice examination are taken from this exercise. If you have difficulty, consult your tutor to discuss the problem.
  5. Complete the practice examination provided for this unit. Evaluate yourself, first checking your answers against those provided in the Answers section at the end of the textbook, and then comparing your solutions with those provided on pages S-397 to S-410 of the Student Solutions Manual portion of the textbook. The number of points in a question may indicate the number of steps in the solution. Give yourself full credit if your answer is correct and you give a complete solution, even if your solution differs from that shown in the Student Solutions Manual.

Practice Examination

Time: 2 hours
Total points: 50
Passing grade: 50%

Do the following exercises from the Chapter 9 Review Exercise on pages 390–392 of the textbook.

To obtain full credit you must justify all your answers and show your work.

  1. Exercise 6  (Marks: 5 pts.)
  2. Exercise 10  (Marks: 5 pts.)
  3. Exercise 14  (Marks: 6 pts.)
  4. Exercise 16  (Marks: 8 pts.)
  5. Exercise 26  (Marks: 12 pts.)
  6. Exercise 32  (Marks: 4 pts.)
  7. Exercise 38  (Marks: 10 pts.)

TOP