Better Decision-Making Through Collaborative Development of Proposals

Online-participationdecision-makingApplications

Authors: Björn Ebbinghaus, Martin Mauve

PDF DOI

BibTeX
@inproceedings{10.1007/978-3-031-05544-7_2,
	abstract = {In a traditional decision-making process, proposals are made and usually commented on by the participants, and finally a vote is taken. We have found in past public decision-making processes that there are also discussions about negative sides of proposals together with possible improvements of them. We have taken this as an opportunity to model the improvement of proposals within the decision-making process.},
	address = {Cham},
	author = {Ebbinghaus, Bj{\"o}rn and Mauve, Martin},
	booktitle = {HCI in Business, Government and Organizations},
	editor = {Fui-Hoon Nah, Fiona and Siau, Keng},
	isbn = {978-3-031-05544-7},
	pages = {11--23},
	publisher = {Springer International Publishing},
	title = {Better Decision-Making Through Collaborative Development of Proposals},
	year = {2022}
}

Abstract In a traditional decision-making process, proposals are made and usually commented on by the participants, and finally a vote is taken. We have found in past public decision-making processes that there are also discussions about negative sides of proposals together with possible improvements of them. We have taken this as an opportunity to model the improvement of proposals within the decision-making process.

Our new model for this is similar to version control systems like git work. Proposals in a decision process can have none, one or several predecessors. This structure allows different constructs of the real world to be modelled. Where previously only proposals could be made without reference and structure among each other, the system presented in this work allows modelling of further developments of proposals or even coalitions and compromises, in a real-time collaborative decision-making process.

To validate our ideas, we tested them in two controlled experiments and concluded that our modifications are useful.

Table of Contents

1 Introduction

Our goal is to understand how a potentially large group of participants can be supported to conduct collaborative online decision-making processes with a minimum amount of external governing factors. Since there exists a multitude of systems that are regularly used for tasks such as online participation, much of the required functionality is well known. This includes mechanisms for proposing an item, discussing it and voting on alternatives. However, there is one aspect of collaborative online decision-making that is not yet well understood: how can a large group of participants work together to come up with and decide on proposals? In real-world politics, there is a vast array of instruments to refine proposals and form coalitions in order to reach a final decision.

Our idea is to take two of the key underlying elements of these instruments and translate them to an online setting. Namely, improving an existing proposal to gather more support for it and forming a coalition to join the support of two or more proposals. We believe that these elements can be translated to mechanisms that are similar to collaborative software development through distributed version control systems.

Based on this idea, we have developed decide, an application for collaborative online decision-making that supports the participants to work together to come up with the best possible solution for a given issue. This does not require the participants to have homogenous interests and agree on a solution. It gives them all the tools they need to come up with good proposals, improve them, form coalitions and decide on the result.

We have conducted several experiments using decide. The results of those experiments are encouraging (see 4). Participants understand and use the new mechanisms provided for improving and merging proposals. Our experiments also suggest that the specific voting system employed may have a significant impact on how the participants use the option to improve and merge proposals.

The overall idea is inspired by observations of shortcomings in a recent participatory-budgeting process of ours [1, 2] where we asked participants to make proposals on how to improve the study course of computer science at hhu. For that process, we used D-BAS [3], a dialog-based argumentation system, to collect proposals and let participants argue about them. They were able to present pro- and contra-arguments to each other and to help inform newcomers to the process. The proposals made were used in a final vote. In this process, we noticed two behaviours:

During the process, participants made proposals that were very similar. Some proposals were slight amendments of other proposals, or they overlapped in their intentions. This poses a problem in a later vote as it can lead to vote splitting — i.e. votes are split between two candidates where there would have been a majority with only one candidate — or, on the contrary, if both overlapping proposals win at the same time, they cannot be implemented as they represent mutually exclusive issues.

Secondly, we found that when discussing proposals, participants often defended (or attacked) the proposal in question by providing possible implementation details or alternative solutions. They lacked the option of modifying or deleting an existing proposal, so their only option to add modifying details would be to make a completely new and separate proposal, which means more friction in the participation process.

1.1 Related Work

The field of collaborative decision-making has a long history. Most of it, however, deals with formal processes and (semi-)automatic decisions, often in the context of business.

This paper deals with a mainly social domain. It does not present a new decision-making framework, but an extension of traditional participatory processes in which proposals are made by participants, accompanied by a software prototype.

With the ever-increasing availability and use of the internet, online participation processes have become increasingly popular, sometimes involving large segments of the population. From informal opinion polls to legally binding participatory budgets.

In a traditional participation process, where proposals can be made by the population, these proposals are submitted by the participants and, often after a review by moderators, are presented to the public. Afterwards, there is the possibility to comment on these proposals and possibly already give one’s approval. This can be possible for everyone who visits the respective website in order to keep the hurdle to participation low, or it is secured via postal invitation codes or e-passports; this usually only happens when legally binding decisions are involved.

A similar real world process is called consensus decision-making or proposal-based decision-making, which is similar to our system in that it focuses on the development of proposals, but since it is offline, it is synchronous, meaning all participants collaborate on the same step in the decision-making process at once. This is simply not feasibly for large, online processes.

1.2 Structure

In this paper we will first explain the core idea of this paper and how it has been integrated into a complete decision-making process.

We will then briefly look at voting methods, as these are relevant to the way participants interact with the system and are then used to make the final decision. Among other things, we have learned how much influence the voting method has on the motivation to cooperate and the perception of fairness.

We then present how we have experimentally tested our idea for feasibility. In doing so, we consider our idea as part of the complete system. We also explain and discuss the observations, conclusions and results.

Finally, we will draw a conclusion and show what further steps can be taken with this work.

2 The decide Decision-Making Platform

As a proof of concept for our idea, we developed decide. A web application, tailored for groups of non-expert participants to develop proposals together, and eventually make a decision.1

A decision-making process starts with someone putting forward a topic for decision and defining certain parameters, such as the desired time frame, restrictions on certain participants and visibility. Care should be taken here to describe the topic as precisely as possible and to do justice to the issue to be addressed. A decision-making process with a handful of friends will have different characteristics than a decision-making process of governmental and administrative institutions with an indefinite, large number of participants. Although our idea itself is independent of the number of participants, it is intended for use in larger groups of individual participants. Even though we use terms like coalition and compromise, it does not mean a coalition between two fixed, defined groups such as parties or other interest groups. We aim at grassroots democratic and self-regulated processes.

After a process has started, participants interact with the system by making proposals to solve the issue, arguing about them and at the same time giving their agreement to the proposals made. These approvals serve both as indications during the process of which proposals are popular and where it is worth working together (see 3), and as votes for the final decision, if it is desired that the process has a fixed end at all.

2.1 The Inner Workings

We base the system of the notion of a proposal. A proposal is a text entry made by a participant. Every participant can make a proposal at any time during a process.

Important to note is, that proposals are immutable, no one, not even the authors can modify or even remove them. In this sense the author of a proposal hosts no special power over this proposal, nor is there a notion of stewardship. We even hid the author of a proposal, after observations of internal tests in our group, as not to associate a proposal with a specific person. This way, we foster a sense of belonging to the idea of a proposal rather than to the author. Having immutable proposals is therefore not just a convenience, but a requirement.

Another reason why proposals cannot change is our live voting system. In order to develop proposals, we need a way to communicate interest and approval of proposals between the participants. Therefore, we decided that having a traditional vote at the end of a participation process is not sufficient. Until the end of such a process it would not be visible how much interest and approval there is in a proposal. Approvals in our application are therefore possible and publicly visible to every other participant immediately after the creation of a proposal. This allows participants to assess the current climate of a process and make better decisions about which proposals are worth to develop further. We explain our voting system in more detail in 3.

  1. The decide application is open-source, and the code can be found here: https://link.cs.hhu.de/decide3-repository. We also have an in-development live instance here: https://link.cs.hhu.de/decide3-repository/decide3↩︎

2.2 Processes

Every proposal belongs to exactly one (decision-)process. A process is a topic that frames a decision. It has a title and a description. Optionally there can be a start and/or an end time in which a process is active and participants can participate. Usually there is at least an end time for when a decision process has to come to an end, but nothing speaks against a process that continues forever with ever evolving proposals.

Processes can be public or private and only be visible for invited participants. They can also modify rules about the process, like how much of the approvals are visible. Either the number of approvals is not visible, shown as the sum (this is the default) or with total transparency where everyone can see exactly who approves what. The last setting was popular for more informal decision-processes in small groups.

2.3 Development of Proposals

As we explained, we want to support decision-making by developing a set of proposals to a greater, better set of proposals. For this we allow and encourage multiple ways of adding new proposals.

First, every participant can submit a new proposal with an informative and unique title and a detailed description. In the best case a proposal is solely identifiable by the title, with the description only serving as implementation details or clarification.

In addition to this there are multiple ways of creating new proposals from existing proposals. They are all based on the idea that we want to model the relationship of a new proposal and the proposals that have led to it. [fig:new-proposal] shows one way, how this is presented to participants.

Step by step interface to add a new proposal

We call proposals with a single parent fork and proposals with multiple parents merge. These are terms from version control software, where the state of a project is tracked by a chain of annotated changes. This allows to track every development that ever occurred in the said project. 2 portrays these relationships.

Semantically, a fork can represent a variety of relationships. It would represent an enhancement of a proposal, an alternative or really any related semantic modification of a proposal. This is the intended way of modifying a proposal. Modifying it in place would not be possible as every proposal is a candidate in an ongoing vote. For the moment, we leave the exact meaning to the author of the new proposal. Of course, a more specific annotation would be conceivable, but for now we want to focus on different groups, according to the underlying system.

Merges, proposals with two or more parents, are the construct with the most potential in our system for collaboration. By enabling participants to link multiple proposals into a new one, real-world constructs are made possible. These can be, for example, compromises and coalitions. It should be noted that the content of a new merge proposal is not only a technical linkage of existing proposals. In terms of content, an independent proposal is created here, with its own title, content and arguments. This is a necessary freedom, as participants should not be restricted by the system as to how their merges should be. The application should only play an advisory role and not restrict the participants.

3 Voting

To come to a final decision, we use votes. So far, we already have the advantage that our system is designed to put up more candidates for a voting, which allows participants to better reflect their opinion. However, we have some special requirements in our system. In order for the participants to be able to work effectively with each other, it is necessary for them to be able to communicate amongst themselves in one way or another. We offer three different ways to do this in our system:

  1. The creation of proposals itself (portrayed in 2)

  2. Giving approval to one (or more) proposals

  3. Nested comments on each proposal

This section deals exclusively with Point 2. It is necessary for the participants to recognize during the ongoing process which proposals are liked and which are disliked. Only then will participants be able to suggest useful enhancements and coalitions.

We have looked at different voting methods and weighed up which is suitable for the processes. For this, we have established the following requirements.

  1. It must be easy to cast a vote.

  2. It must be possible to see how popular a proposal is compared to another proposal.

  3. The current ranking of the proposals (and thus also the current winner) must be transparent.

We first used approval voting [4]. It is a good match regarding our requirements. By today most people are familiar with a notion of a like-Button. Just a single click is necessary to cast a vote, fulfilling the first requirement. It is easy to aggregate and display the votes by summing them and thus communicate the current popularity of a proposal. Fulfilling the second requirement, it allows sorting the proposal by this sum, which fulfils the third requirement. This leaves the possibility of a draw, especially in processes with fewer participants. We have decided not to consider this problem further, as draws are an anomaly that can — and often should — be solved in the context of the situation. For our experiments we reliably broke the tie with the creation time of the proposals, letting more recent proposals win over older proposals. We decided on the basis of a gut feeling that newer proposals had less time to convince and woo away potential voters.

We also considered more complex voting methods. In particular, a preference-based voting method, such as irv [5]. In such a method participants have to enter more precise votes. They have to rank proposals against each other. The winner is determined by repeatedly eliminating the worst candidate and recounting the remaining votes, until only the winner remains. While this may result in a vote that better matches their real intentions, it also requires more effort from the participants, especially if the process involves many proposals. Furthermore, with preference voting, it is no longer possible to display a simple aggregate of votes of the other participants. This makes it difficult to compare two proposals and thus the overall state of the process is no longer easily tractable.

The fact, that voting is iterative [6] in our system creates unique situations. While the effect of one’s own vote in a traditional election is only visible after the counting, the participants in our system see the effects immediately.

This means that, for example, strategic voting can be experienced first-hand by participants. Participants can experiment with how they cast their votes and immediately see the impact this has on the current interim result of the process. In a process that uses plurality voting, i.e. a process where only one proposal can be approved, participants immediately see that giving their vote to a proposal has a direct negative effect on their previous favourite. We will see in 4 that this quickly leads participants to perceive the voting method as unfair, a perception that is in line with reality but usually only noticeable to people who have studied the voting method more closely.

Although in this case the weaknesses of the voting method are made more tangible by the immediate feedback, there are also possible advantages. Participants experience the direct influence of their vote, which may motivate a more detailed, further engagement with the candidates.

4 Experiments

To evaluate our platform, we tested our idea in two experiments with untrained students. After the first run, we made modifications based on our observations and feedback from the participants and then reviewed them in the second run. This led to a significant improvement in some parts of the surveys following the experiments.

4.1 Environment

We conducted the two decision-making processes with a duration of five days each, with a subsequent survey via a questionnaire. Both processes had the same theme namely: How can we improve digitalization at the university?. This is a current topic, which affects students from different study programmes and even staff. It lends itself well as a topic for our experiments, as there is still much need for development here. Large-scale digitalization of various aspects of university life is still uncharted territory and therefore lends itself well to input and ideas from students and staff.

In addition to payment for participation, as a special incentive, it was promised that we would forward the most approved proposal to the university’s Pro-Rector for Digitalization, who would then comment on the proposal and the arguments relating to it. We then recruited the participants through advertising in lectures, the student council and personal letters. The majority of participants came from the Faculty of Natural Sciences at our university.

4.2 Setup and Execution

Both groups had the same topic and the same length for their run. Before starting the experiments, the participants were informed about what features the application provides and what the special features are. However, the task was deliberately kept simple. The participants were asked to visit the application daily, learn about new developments and participate in the process. No guidelines were given on how or when to participate. In particular, it was not required that the novel features of the application had to be used.

The processes took place completely remotely, partly due to the current pandemic. Although it is more difficult to observe the process and the participants, the process is closer to reality.

The processes started on Monday at noon. On Wednesday, there was another appeal and a brief overview of the current progress. On Friday at noon, the process ended and we released the questionnaire. At least 15 participants took part in each of the processes, although in both cases only 10 participants were actually active during the entire week like planned.

During the processes we fixed software and minor quality-of-life issues, like improved loading times and better contrast for people with impaired eyesight, based on participant feedback and observations, but the rules and content of the processes were not touched.

In the first experiment, participants had only one approval they could give to a proposal. If they agreed to another proposal, the approval to the former proposal was removed. We decided to do this because we noticed in internal, technical tests that participants tended to just agree to everything. The following working hypothesis was formulated for this:

Hypothesis 1. Some pressure exerted on participants by the decision-making mechanism motivates collaboration with other participants.

Participants of the second experiment could approve as many proposals as they liked. In both cases the winner with the most votes won.

4.3 Observations in the First Experiment

This made for a more active process, as participants had to think more about which proposal they wanted to vote for and which proposals they did not. They could not just approve to everything, a behaviour we noticed in earlier, internal tests.

It is possible for us to track the changes in votes in order to understand the voting behaviour of the participants. This allows us to observe and answer questions such as: Do participants stay in one strand of proposals or do they switch back and forth?, Are participants more likely to shift their approvals to newer, developed proposals?.

Based on the final survey and feedback during the process, we noted that some participants have found the voting procedure unfair or dissatisfactory, as they have had to take a risk with their vote to support a new proposal they like better. Hurting their current favourite proposal by giving their only approval to the new proposal with fewer votes.

Although vote splitting is not a new problem, in our public, live voting system it is directly experienced by the participants and thus poses a bigger problem. An open, live voting system makes strategic voting more visible and thus also the advantages and the disadvantages it entails.

We have thus made the following hypothesis for the next iteration of our system:

Hypothesis 2. The voting mechanism is highly influential to the satisfaction with the decision result in our system.

Based on this knowledge we ran the second experiment with simple approval voting. Each participant can give as many or as few approvals as they want. Approval voting, which is a cardinal voting method, is immune against vote splitting [7]. This removes the risk for participants to vote for a new proposal that has not yet received much attention.

4.4 Observed Differences in the Use of Our New Features

3 shows the proposals and relations between parents and children that emerged during each experiment. It can be seen that in the first experiment much more use of our tools to develop proposals was made, with the same number of participants and proposals.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 Time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Time 15

We therefore assume that our original 1, that more collaboration takes place with a little pressure, is pointing in the right direction. Although this behaviour can be attributed to chance with the low numbers of participants, we believe that the difference in behaviours is significant enough to support our hypothesis.

4.5 Survey Results

The final survey that both groups completed after their process, consists of several sections with different statements. Participants should indicate how much they agree with these statements. We used a Likert scale [8] from 1 (strongly disagree) to 5 (strongly agree).2

The topics of the sections are:

  1. First encounter with the application.

  2. Feelings regarding the outcome.

  3. Statements about the process in general.

  4. Statements about argumentation.

  5. Statements about forks.

  6. Statements about merges.

Both groups do not differ significantly in their answers except for statements in one section. Overall the reception to the experiment, the application and the features it comes with was positive. The biggest criticism of the application by the participants was the confusing nature given a larger number of proposals or arguments. There was a wish for clearer indications of which proposals came from which other proposals.

Questionnaire-Section [item:q2], about the participants feelings regarding the outcome, had significantly more positive grades with the second experiment, where participants had unlimited approvals. We compared the statements of both groups with a two-sided t-test for different variances [9], due to our low n of 10 and 12 respectively. The values are presented in 1.

Comparison the statements in question-section [item:q2] of the first experiment with a single approval and second experiment with multiple approvals. 1 = strongly disagree, 5 = strongly agree
Plurality Approval
Statement n Avg. Var. n Avg. Var. p(TT)
I am satisfied with the outcome of the decision. 12 3,75 0,75 10 4,70 0,23 0,005
I think my opinion contributed to the final decision. 12 3,50 0,64 10 4,60 0,49 0,003
I find my opinion reflected in the decision. 12 3,67 1,15 10 4,70 0,23 0,009
I think my voice was heard. 12 4,00 0,73 10 4,80 0,18 0,011
I felt the process was fair. 12 4,00 1,64 10 4,80 0,18 0,061
I think I was correctly informed about the process. 12 4,50 0,45 10 4,70 0,90 0,584
I think the decision is a joint product of everyone. 12 3,75 1,48 10 4,80 0,18 0,014

The clear difference in the results of the two experimental groups reinforces our belief that the perceived fairness of a voting system used is a key determinant of satisfaction with the process and outcome of a process in our application. We accept 2. This should therefore be considered and investigated in further research.

  1. The full, translated data, can be found here: https://link.cs.hhu.de/data-ebbinghaus22_collabproposal.↩︎

5 Conclusion and Future Work

In this paper, we have shown a system with features that provide participants in a decision-making process with tools for developing proposals together.

We have shown that these features are understood and used by participants. However, we have also learned through the second experiment that such a system, in its current state, needs to encourage participants to collaborate.

In our first experiment, this happened involuntarily due to constraints in the voting method. Participants were motivated by the characteristics of the voting method not to deviate further from the currently strong proposals and therefore tend to stick with the existing ones. By being limited to one approval, exploring new proposals became a risk for participants. Although effective, we believe this is not good for future processes because it goes against our overarching goal of providing the tools to make more and better proposals.

A suitable methodology to encourage participants to cooperate and at the same time provide a voting process that is perceived as fair is thus indispensable for further research on this system.

We believe that the system itself could provide more guidance and advice to participants in a process. Currently, the system itself is not smart, it does not actively support participants, e.g. by recommending suitable proposals or worthwhile coalitions. We have already made first attempts in this direction. By using a public, iterative voting method, the system already has information about which voter groups support which proposals during a decision-making process. This can be used for collaborative filtering, among other things, in the future.

Acknowledgements

Björn Ebbinghaus is a member of the PhD-programme Online Participation, supported by the North Rhine-Westphalian funding scheme Forschungskollegs.

1.
Ebbinghaus, B.: Decision Making with Argumentation Graphs, (2019). https://doi.org/10.13140/RG.2.2.12515.09760/1.
2.
Ebbinghaus, B., Mauve, M.: decide: Supporting Participatory Budgeting with Online Argumentation. In: Prakken, H., Bistarelli, S., Francesco, S., and Taticchi, C. (eds.) Computational Models of Argument. Proceedings of COMMA 2020. pp. 463–464. IOS Press (2020). https://doi.org/10.3233/FAIA200535.
3.
Krauthoff, T., Meter, C., Betz, G., Baurmann, M., Mauve, M.: D-BAS – A Dialog-Based Online Argumentation System. In: Computational models of argument. pp. 325–336, Warsaw (2018). https://doi.org/10.3233/978-1-61499-906-5-325.
4.
Weber, R.J.: Approval voting. Journal of Economic Perspectives. 9, 39–49 (1995).
5.
Brandt, F., Conitzer, V., Endriss, U., Lang, J., Procaccia, A.D.: Handbook of computational social choice. Cambridge University Press (2016).
6.
Meir, R.: Iterative voting. In: Endriss, U. (ed.) Trends in computational social choice. pp. 69–86. AI Access (2017).
7.
Poundstone, W.: Gaming the vote: Why elections aren’t fair (and what we can do about it). Macmillan (2009).
8.
Likert, R.: A technique for the measurement of attitudes. Archives of psychology. (1932).
9.
Welch, B.L.: The generalization of ‘STUDENT’s’problem when several different population varlances are involved. Biometrika. 34, 28–35 (1947).