----start---- class of 2000 epidemiolgy 2/13/98 lecturer: David T. Galligan transcriber: me handout: decision analysis: contains slides. Decision Analysis all slides are in the handout! why is decision analysis important? You'll spend the rest of your life making them. there are certain principles which, once you understand them, will help you make better decisions. This lecture should help you understand the fundamentals. how do we make decisions? information--> analysis-->decision based on a criteria we take information, analyze it, and make a decision based on our analysis. hopefully your education allows you to do a better job with your analysis than your client, who comes to you for help with making a decision. examples of information would be: age of animal, results of PE, lab test results, physiological parameters, etc. you then process the information and make an analysis: figure out the animal's probability of responding, given the information you have. you should know that various treatments have various probabilities of success. the criteria you use to make your decision may be animal survival, minimum cost, maximum profit, minimal risk, etc. Some problems may have multiple criteria, and you would evaluate them under different dimensions. structure of decisions: choices. how do we structure our decisions? the basic accepted structure is the identification of the things the decisionmaker has control of. look at the process and say "what can I control?" eg, treat or do not treat. use tx A or B. Use this test, or don't use it. Decision choices are schematically represented as square boxes. note that as litigation increases, people are using tests more and more - this isn't appropriate for all food animal situations. the key concept is that you represent a choice as a box, and that it is something you have control over. chance events: things decisionmaker has no control over: does animal live or die? what is the level of response to the treatment? what's the outcome of a test? how accurate is the test? you represent these chance events as a circle. what happens is, you define a decision, or structure it rather, in terms of chance nodes and decision nodes, and their time course of occurrence. you make a decision to run test before you experience the chance node of the results. so decision analysis involves structuring analysis in terms of its temporal pattern. outcomes: third element of decisions. choice, chance, then outcomes. basically the outcomes are what states are the animals going to move to as a result or consequence of your decision or lack of decision? it can be favorable, unfavorable, live, die, pregnant, open, etc. you put a value on those outcome results. value might be cost, profit, or whatever. simple example: a cow is presented with an LDA And you must decide whether to do surgery or a toggle intervention. generally with LDA you either cull the cow, do a ventral midline surgical approach, or a toggle/mollybolt procedure. you have to structure a decision as to how to proceed. first: gather information. what information? value of healthy cow - what she's worth if she responds value of unhealthy cow - what's the salvage value if she doesn't do well? cost of surgery cost of toggle probability of success of toggle probability of success of surgery - how do you get the probability information? analyze your past experiences, make a guess (subjective probability - useful, necessary often) find out goal of producer - to maximize value of animal, to control cost, etc. construction of the decision: choices: the cow is in front of you and definitely has an LDA. the first thing you have to do is choose between doing the surgery or a toggle. this shows up on your chart as a box - you have control over this. furthermore, you represent the alternatives by lines coming out of the box. a third alternative would just be a third line coming out of the box, labelled "cull". next phase: identify chance outcomes. if you do surgery, you have a chance of a good or bad outcome. if you do a toggle, you have a chance of a good or bad outcome. if you cull, you have no chance event. sometimes you have a decision box, a chance node, and then another decision box. these schematics can take on a tree look, where the various decision choices and chance events are the arboration. also the temporal nature is represented by a left to right timeline structure. probability values: now we assign probability values for each of the chance events. all must sum to one over a given chance node. you have to define all possible outcomes and their frequency. so say that surgery gives 85% good outcomes and 15% bad outcomes. the toggle gives a 25% good outcome and a 75% bad outcome. for each event, the sum of the probability is one. outcome values: we have to define these, now. say the outcome of a good result is that the cow is worth $1000, and a bad result causes the cow to be worth $400. now the whole tree is structured. the next step is to decide which intervention to pursue - to solve the decision tree. how? by folding back the tree. we have to also first assign cost to the interventions. the surgery has to be paid for - it's $120 regardless of outcome. the toggle is $30. you have to subtract these amounts from the outcome values. therfore, a good response to surgery is really worth $1000 - $120, for example. NOW we fold back the decision tree. this is analysing it. look at each chance node and ask what's the expected value? multiply the probability times the outcome value. so .85 (1000-120) + .15 (400-120) = $790. the expected value is the weighted average. for the toggle, .25(1000-30) + .75(400-30) = 520. so, which branch do you want to pursue? the surgical branch has an average outcome of $790 nd the toggle branch has an average outcome of $520. most producers would want the surgical method! the structure of this lends itself to other problems. with small animals, it would be living or dying (represented as 1 or 0 for the outcome value) the next phase, perhaps the most important part, is that people might argue with the numbers you put in. maybe they'll say a toggle should have a higher success rate or something. you might want to do a sensitivity analysis - vary a parameter over a likely range you might expect, and see at what point the other decision is more appropriate. this is most useful when you've made a subjective decision. sensitivity analysis - probability of success with toggle. it can go from 0 to 1. you set a value for each possible value. before, it was .25 and we had an outcome value of 520. if probability is 0, outcome value becomes $370. if probability of success is 1, the expected value is $970. at .7 probability of success of toggle, expected value is the same as surgical intervention's expected value. however, you have to now decide what the likelihood of a .7 probability of success of the toggle is. you can make a sensitivity graph instead of a table - as probability of success increases, so does the expected value. then you can graph the expected value of surgical intervention on the same chart, and you see where the lines cross - which is at .7. this is on p 4 of the handout. what you'll find is that when you apply this to different problems, you may have many different choices, not always just one intervention or the other. similarly, you could vary the probability of success for surgery and see at what point you'd want to do the toggle. want to choose the option with the higher expected return value. someone now might say well, what happens when there are other things that are varying? probability of success of toggle and surgery might both vary at the same time. well, then you have to do a multiway sensitivity. there, you look at what are the conditions when the expected value of surgery equals the expected value of toggle. when are the conditions equal in expected value? set the equations equal to each other. P1 * value favorable + (1-P1)(value unfavorable) - cost 1 = P2* value favorable + value favorable - value unfavorable = cost1-cost2 / P1-P2 (note to self - see handout for equations) so we get the cost difference, which is fixed at $90. then figure out what are the possible variations of P1 and P2? that will always vary from 0 to 1. you can use this equation somehow to make a chart plotting the value vs the probability. so if the difference in values is $500, the difference in probability has to be somewhere about .25-.27. if the difference is greater than that, do the more expensive intervention, and if it's less, do the less costly intervention. above the curve you do surgery, below the curve you do the toggle. as difference in value gets lower, you are more likely to choose the less expensive interventions. if cull value is closer to good value, you do the cheaper procedure. well, what's happening in vet med? animals are getting cheap. value favorable and unfavorable are both decreasing - so we're doing mroe cheap interventions despite their lower success rates. what happens to the line if we look at a more expensive intervention? you do fewer of them. the line moves up and to the right, increasing the number of low cost interventions. with the rise in health care costs, this is also occuring.see the multiway sensitivity charts in the handout. now, once you've made this kind of chart, you can apply it to other problems. often what happens is the value of these isn't so much what you learn for a specific application but more for what should your strategy be formaking decisons. this is multiway sensitivity. you can read a JAVMA article if you want. on exam: simple decision tree with probabilities you have to fold back and analyze, and a sensitivity chart. P1 is probability of success of surgery, P2 is probability of success of surgery. why can you do this? because you're setting the interventions equal to eachother. the line represents the conditions when the two interventions are equal in value. which would you choose? intervention 1: good outcome + $4000; bad outcome -$2000 intervention 2: good outcome +$2000; bad +$1000 which is better? would you rather take an exam that you can get a B or C on, or an A or D on? well, what are the probabilities? well, what if it's 50-50? well, how many times can you play the game? at a herd level, you play a lot. suppose it's .75 good and .25 bad for 1, and 50 50 for 2. which would you choose? 1. on an expected value basis it comes out to 2500 for 1, and 1500 for 2 - but for option one, you might end up with a negative value. so most people would choose 1 if they can play many times, but if they're only playing once, they might choose 2. intervention 1 is called a "risky" intervention. you have to look at the risk. the concept of risk - extreme results, extreme values. the concept of playing the game many times - usually if you do you will end up with the expected values. for a young sire, there might be more variation in outcome vs a proven sire - therefore young sire is more risky. the last thing - we're goign to discuss decision trees using test information in the next hour. -------break---- ok. i have to do note service on this part. we're going to do an example of the last part of last lecture: value of healthy cow: $1500 value of salvage: $500 success of intervention 1 is 80% success of 2 is 60% cost of 1 is $300 cost of 2 is $210 make the chart as shown. * if you do the intervention, as described above, your point ends up at 0.2 on the X and 1000 on the Y - and that is in the "intervention one" area. so you do intervention one.as the cost of animals gets lower and lower, more of the time we have to choose the intervention at the bottom left of the chart. Milk Progesterone as a Test for Pregnancy: decisions with test information this is going to show us how to use test information. when you think about this - this is what you do. take HR, temp, interpret that, make decisions. they did a survey of vets and how they used test information and how they analyzed it, and 90% of vets drew improper conclusions from their tests. This is because they didn't understand basic principles of decision analysis. HOpefully we can learn it now. we'll use an example based on this milk progesterone test for pregnancy. there was a paper written on using this test instead of rectally palpating the cow. normally, cow progesterone levels fluctuate during the heat cycle (21 d). there are low levels of progesterone during estrus. when she becomes pregnant, she has a high level of progesterone. if you breed her, then test her 21 days later, if progesterone is high, maybe that means she's pregnant, and if it's low, maybe she's coming into heat, and not pregnant. does this test work? they use a test where you squirt milk into a well - if progesterone is high it stays white, if low, it turns blue. how does this relate to rebreeding a cow? what's the value of the test to the producer? what we see here is a graph showing two conditions: that where we don't use the test, and where we do. we breed them all, and at 21 days if they're off the system, we don't know which are open and which are pregnant. we just wait 21 days, check them manually at day 42, then rebreed any open cows. that's what you do if you're not using the test. say you do use the test. you breed all the cows. 21 days later you do the test. if the test is blue, she's not pregnant, so you rebreed them now, saving 21 days. now, some of those cows aren't really open even though the test says they are. you're going to rebreed those too, so there's some error causing you to overbreed. the benefit of the test is capturing 21 days earlier breeding on a proportion of cows. depending on your herd, proportions of animals in each group will vary. the point is that you can ID open cows 21 days early, and breed them. if you get some of them pregnant, they're pregnant earlier than they otherwise would be. we're still rectalling all cows at 42 days. how do you make your decision tree? the thing to decide is: use the test, or not use it. that's your first square. if you don't use the test, you have a chance event at day 42 - some will be pregnant,and some will be open. so that's your circle on the bottom branch of the tree. if you do use the test, it will be positive or negative by chance. that's your first circle. if it's positive, you do nothing til day 42, then at day 42 some will be pregnant and some will be open. depending on accuracy of the test, which you can't control, so it's a circle. if the test is negative, you rebreed, and a proportion of those cows will really be pregnant, and some will be open, and we'll check them at day 42 and rebreed them. so how do you solve this tree and fold it back? the outcome values are listed on the right side of the diagram. these are on a cost basis. when you use the test and it's positive and correct, the cost is the cost of the test RMPA + 0 days - this is our baseline. if you don't use the test, at 42 days, if she's pregnant, the cost is simply 0 days. no test cost. if she's open, you've lost 42 days on her. so her cost is 42 days. when you do the test, if it's negative, and you rebreed her and she's actually alraedy pregnant, your cost is the RMPA + breeding cost. but some of them were really open, and of those, some get pregnant and some stay open. the ones that get pregnant cost us RMPA + 21 days, and those that are still open cost RMPA + 42 days. this is pretty complex. you can try to mimic real decision processes with these trees, pretty well. let's look at dollar values - in the next diagram. the cost of RMPA is $6, the cost of an open day is $2. if test says she's pregnant and she's really open, our cost is $90. if it says she isn't pregnant, and she is, our cost is $16. if it says she's not pregnant, and she isn't, and you rebreed, and she's still not, you cost $90.this is all in the chart. now, we have to figure out probabilities. how? get some information about the test and the breeding process: conception rate: 60% of cows normally get pregnant after the first breeding cost of Day Open: $2 cost of RMPA: $6 sensitivity: 85% specificity: 95% breeding: $10 remember: sensitivity and specificity are attributes of the test, measured against a gold standard. now, get the baseline test information: make a 2x2 table test pregnant open + 85 (sn) 5 - 15 95 (sp) total 100 100 so, we have a 200 cow herd. 100 of them are pregnant, using this scenario. that's 50% of them. *but* the real prevalence of pregnancy is 60%. so you have to convert this table to reflect the real prevalence. so we multiply all the numbers in the pregnant column by 0.6, and all the numbers in the open column by 0.4 in order to adjust for prevalence. now we have: test pregnant open + 51 2 - 9 38 total 60 40 now we have a table reflecting the real herd. this is a "joint probability table". realize this proves sensitivity and specificity are independent of prevalence. it's this joint probability table that people fail to calculate. another name for this is Bay's theorem, or something like that. the probabilities we've calculated are called "joint probability" - the probability of having the test and the outcome. the probability of a cow being pregnant and hving a positive test is 51%. this is different from the probability that if a cow is known to be pregnant, her test will be positive. that's sensitivity, and that's a conditional probability, which is conditional on known outcome values. now, we take that table and we add across horizontally: test pregnant open + 51 2 53 - 9 38 47 total 60 40 100 the bottom row and the right column are called "marginals" from a joint probability table we can calculate sensitivity, specificity, predictive values - what were they? they were based on test outcome, not outcome of the condition but is the test pos or neg,and what proportion of those cows are positive? given a + result, how mny of them are pregnant? well, 53 test +, and 51 are really pregnant. 51/53 is the % of cows that are pregnant, and is the PPV. that's what you use in decision making. that's the probability you put in the decision tree. it may be different from sensitivity - that was 85%, but PPV is closer to 96%. furthermore, the PVs are dependent on the underlying prevalence of the condition in the herd - remember we multiplied everything by the prevalence. so, many of you will look at test information and figure if it's positive it's positive - but you must consider the population from which the subject of the test was drawn! so our PPV is 51/53 = 96% negPPV is 38/47 = 81% realize these are conditional on test results, whereas sens and spec are dependent on outcomes. that's the main difference. given that a cow has a positive test, her probability of being pregnant is 96%. question: a cow is presented with a positive test from a herd with an underlying conception rate of 40%. what's her probability of being pregnant? figure it out for homework. ** he highly recommends that you do this ** it's really important. go back to your decision tree. fill out the probabilities. some of them are the marginal values - the frequency of testing + or - were .53 and .47, remember. for cows we don't test, probabilities equal the underlying prevalence - .6 and .4 final outcome of breeding event is always .6 and .4 of animals that had a positive test, what proportion of those are pregnant? look at your PPV. our PPV was 96% only 4% of cows that test + are really open. so you fill that in for the probability on the top right of the tree. then, giving that the test is negative, .19 are really pregnant, and .81 are really negative. these are your negative predictive values. you need to know this. there are some fundamental principles you learn in vet school and this is one of them - like the Krebs cycle (yeah, like we know that??). go back to the table of test attributes. 85% of the time if an animal is pregnant, she's goign to test positive. 95% of the time if she's open she'll test open. our negative predictive value is 38 truly open cows out of 47 total cows that test negative, and that's 81%. now go back to the tree. now we have to fold back the tree. start at the top. what's the expected value of the top right chance node? 0.96*6 + .04*90 = 9.36 do the same thing for the middle right chance node - that comes out to 64.8 the bottom branch comes out to 33.60 - that's what you pay per cow without using the test. if you get more than 60% pregnant, that number drops. now we have to move back. the top left chance node comes out to be $31.05/cow. this is less than 33.60 - so you'd do better using the test. please referto the handout with all the numbers to see all the equations. sensitivity analysis: if conception rate goes up to 77% we won't use the test, because the test won't capture enough benefit - most of them are pregnant anyway. if the cost of an open day goes down, we won't want to use the test because there's not as much benefit. if cost of test increases, breeding cost changes, etc we have to reassess. you should note that if all the cows are open, your best bet might be to just rebreed everyone, and not do the test. when you do the sensitivity analysis you can plot out a line and see when it ismore appropriate to use the test, and when it isn't. a positive value favors the test. ---end----