Yaqing Hou's Homepage

CEC2024 Competition on “Evolutionary Multi-task Optimization”

Please re-deliver relevant documents to this email address: 2024mtocompetition@gmail.com

OVERVIEW AND AIMS

The original inspiration of artificial intelligence (AI) was to build autonomous systems that were capable of demonstrating human-like behaviours within certain application areas. However, given the present-day data deluge, rapid increase in computational resources, and key improvements in machine learning algorithms, modern AI systems have begun to far exceed humanly achievable performance across a variety of domains. Some well known examples of this reality include IBM Watson winning Jeopardy!, and Google DeepMind’s AlphaGo beating the world’s leading Go player. Given such advances, it is deemed that what we foresee for AI in the future need no longer be limited to an anthropomorphic vision. Indeed, it may be more meaningful to build AI systems that complement and augment human intelligence, excelling at those tasks for which humans are ill-equipped. In this regard, one of the long-standing goals of AI has been to effectively multitask; i.e., learning to solve many tasks at the same time [1]. It is worth noting that although humans are generally unable to tackle multiple problems simultaneously, or within short timespans – as interleaving more than one task usually entails a considerable switching cost during which the brain must readjust from one to the other – machines are largely free from such computational bottlenecks. Thus, not only can machines move more fluidly between tasks, but, when related tasks are bundled together, it is also possible for them to seamlessly transfer data (encapsulating some problem-solving knowledge) among them. As a result, while an AI attempts to solve a complex task, several other simpler ones may be unintentionally solved. Moreover, the knowledge learned unintentionally can then be harnessed for intentional use.

In line with the above, evolutionary multitasking is an emerging concept in computational intelligence that realises the theme of efficient multi-task problem-solving in the domain of numerical optimization [2-6] and practical applications [7]. It is worth noting that in the natural world, the process of evolution has, in a single run, successfully produced diverse living organisms that are skilled at survival in a variety of ecological niches. In other words, the process of evolution can itself be thought of as a massive multi-task engine where each niche forms a task in an otherwise complex multifaceted fitness landscape, and the population of all living organisms is simultaneously evolving to survive in one niche or the other. Interestingly, it may happen that the genetic material evolved for one task is effective for another as well, in which case the scope for inter-task genetic transfers facilitates frequent leaps in the evolutionary progression towards superior individuals. Being nature-inspired optimisation procedures, it has recently been shown that evolutionary algorithms (EAs) are not only equipped to mimic Darwinian principles of “survival-of-the-fittest”, but their reproduction operators are also capable of inducing the afore-stated inter-task genetic transfers in multitask optimisation settings; although, the practical implications of the latter are yet to be fully studied and exploited in the literature. The potential efficacy of this new perspective, as opposed to traditional approaches of solving each optimisation problem in isolation, has nevertheless been unveiled by so-called multi-factorial EAs (MFEAs).

Evolutionary multitasking opens up new horizons for researchers in the field of evolutionary computation. It provides a promising means to deal with the ever-increasing number, variety and complexity of optimisation tasks. More importantly, rapid advances in cloud computing could eventually turn optimisation into an on-demand service hosted on the cloud. In such a case, a variety of optimisation tasks would be simultaneously executed by the service engine where evolutionary multitasking may harness the underlying synergy between multiple tasks to provide consumers with faster and better solutions.

COMPETITION BONUS: 10000RMB(≈$1400)

TEST SUITES
Single-objective and multi-objective continuous optimization have been intensively studied in the community of evolutionary optimization where many well-known test suites are available. As a preliminary attempt, we have designed two MTO test suites [8],[9] for single-objective and multi-objective continuous optimization tasks, respectively.

The test suite for multi-task single-objective optimization (MTSOO) contains nine MTO complex problems, and ten 50-task MTO benchmark problems. Each of the complex MTO problem consists of two single-objective continuous optimization tasks, while each of the 50-task MTO problem contains 50 single-objective continuous optimization tasks, which bear certain commonality and complementarity in terms of the global optimum and the fitness landscape. These MTO problems possess different degrees of latent synergy between their involved component tasks.

The test suite for multi-task multi-objective optimization (MTMOO) includes nine MTO complex problems, and ten 50-task MTO benchmark problems. Each of the complex MTO problem consists of two multi-objective continuous optimization tasks, while each of the 50-task MTO problem contains 50 multi-objective continuous optimization tasks, which bear certain commonality and complementarity in terms of the Pareto optimal solutions and the fitness landscape. The MTO problems feature different degrees of latent synergy between their involved two component tasks.

All benchmark problems included in these two test suites are elaborated in technical reports [8],[9], respectively. Their associated code in Matlab is downloadable at here.

COMPETITION PROTOCOL
Potential participants in this competition may target at either or both of MTSOO and MTMOO while using all benchmark problems in the corresponding test suites as described above for performance evaluation.

For MTSOO test suite:

(1) Experimental settings
For each of 19 benchmark problems in this test suite, an algorithm is required to be executed for 30 runs where each run should employ different random seeds for the pseudo-random number generator(s) used in the algorithm. Note: It is prohibited to execute multiple 30 runs and deliberately pick up the best one.

For all 2-task benchmark problems, the maximal number of function evaluations (maxFEs) used to terminate an algorithm in a run is set to 200,000, while the maxFEs is set to 5,000,000 for all 50-task benchmark problems. In the multitasking scenario, one function evaluation means calculation of the objective function value of any component task without distinguishing different tasks.

Note: The parameter setting of an algorithm is required to be identical for each benchmark problem in this test suite, respectively. Participants are required to report the used parameter setting for each problem in the final submission to the competition. Please refer to “SUBMISSION GUIDELINE” for more details.

(2) Intermediate results required to be recorded
When an algorithm is executed to solve a specific benchmark problem in a run, the so far achieved best function error value (BFEV) w.r.t. each component task of this problem should be recorded when the current number of function evaluations reaches any of the predefined values which are set to k*maxFEs/Z, (k =1, …, Z; Z=100 for 2-task MTO problems and Z=1000 for 50-task MTO problems), in this competition. BFEV is calculated as the difference between the best objective function value achieved so far and the globally optimal objective function value known in advance. As a result, 100 BFEVs would be recorded for every 2-task benchmark problem, while 1000 BFEVs would be recorded for every 50-task benchmark problem, w.r.t. each component task in each run.

Intermediate results for each benchmark problem are required to be saved separately into nine “.txt” files named as "MTOSOO_P1.txt", …, "MTOSOO_P9.txt" for the nine MTO complex problems, and "MTOMSO_P1.txt", …, "MTOMSO_P10.txt" for the ten 50-task MTO benchmark problems.

1*maxFEs/Z, BFEV_{1,1}^1,…, BFEV_{1,1}^n,……, BFEV_{30,1}^1,…, BFEV_{30,1}^n
2*maxFEs/Z, BFEV_{1,2}^1,…, BFEV_{1,2}^n,……, BFEV_{30,2}^1,…, BFEV_{30,2}^n.
…
m*maxFEs/Z, BFEV_{1,m}^1,…, BFEV_{1,m}^n,……, BFEV_{30,m}^1,…, BFEV_{30,m}^n

where BFEV_{j,k}^i (i = 1, …, n ;j = 1, …, 30; k = 1, …, m) stands for the BFEV w.r.t. the ith component task obtained in the jth run at the kth predefined number of function evaluations. Note: n=2 and m=100 for 2-task benchmark problems, while n=50 and m=1000 for 50-task benchmark problems.

The first column stores the predefined numbers of function evaluations at which intermediate results are recorded. The subsequent columns store intermediate results for each of 30 runs with each run occupying n consecutive columns w.r.t. n component tasks, respectively. Note: The comma is used as a delimiter to separate any two numbers next to each other in a row. As an example, “.txt” files obtained by MFEA are provided as reference.

(3) Overall ranking criterion
To derive the overall ranking for each algorithm participating in the competition, we will take into account of the performance of an algorithm on each component task in each benchmark problem under varying computational budgets from small to large. Specifically, we will treat each component task in each benchmark problem as one individual task, ending up with a total of 518 individual tasks. For each algorithm to be ranked, the median BFEV over 30 runs will be calculated at each checkpoint which corresponds to different computational budgets for each of 518 individual tasks. Based on these calculated data, the overall ranking criterion will be defined. To avoid deliberate calibration of the algorithm to cater for the overall ranking criterion, we will release the formulation of the overall ranking criterion after the competition submission deadline.

For MTMOO test suite:

(1) Experimental settings
For each of 19 benchmark problems in this test suite, an algorithm is required to be executed for 30 runs where each run should employ different random seeds for the pseudo-random number generator(s) used in the algorithm. Note: It is prohibited to execute multiple 30 runs and deliberately pick up the best one.

For all 2-task benchmark problems, the maximal number of function evaluations (maxFEs) used to terminate an algorithm in a run is set to 200,000, while the maxFEs is set to 5,000,000 for all 50-task benchmark problems. In the multitasking scenario, one function evaluation means calculation of the values of multiple objective functions of any component task without distinguishing different tasks.
Note: The parameter setting of an algorithm is required to be identical for each benchmark problem in this test suite, respectively. Participants are required to report the used parameter setting for each problem in the final submission to the competition. Please refer to “SUBMISSION GUIDELINE” for more details.

(2) Intermediate results required to be recorded
When an algorithm is executed to solve a specific benchmark problem in a run, the obtained inverted generational distance (IGD) value w.r.t. each component task of this problem should be recorded when the current number of function evaluations reaches any of the predefined values which are set to k*maxFEs/Z, (k =1, …, Z; Z=100 for 2-task MTO problems and Z=1000 for 50-task MTO problems), in this competition. IGD [10] is a commonly used performance metric in multi-objective optimization to evaluate the quality (convergence and diversity) of the currently obtained Pareto front by comparing it to the optimal Pareto front known in advance. As a result, 100 IGD values would be recorded for every 2-task benchmark problem, while 1000 IGD values would be recorded for every 50-task benchmark problem, w.r.t. each component task in each run.

Intermediate results for each benchmark problem are required to be saved into separate ".txt" files: "MTOMOO_P1.txt", …, "MTOMOO_P9.txt" for the nine MTO complex problems, and "MTOMMO_P1.txt", …, "MTOMMO_P10.txt" for the ten 50-task MTO benchmark problems. The data contained in each ".txt" file must conform to the following format:

1*maxFEs/Z, IGD_{1,1}^1,…, IGD_{1,1}^n,……, IGD_{30,1}^1,…, IGD_{30,1}^n
2*maxFEs/Z, IGD_{1,2}^1,…, IGD_{1,2}^n,……, IGD_{30,2}^1,…, IGD_{30,2}^n
…
m*maxFEs/Z, IGD_{1,m}^1,…, IGD_{1,m}^n,……, IGD_{30,m}^1,…, IGD_{30,m}^n

where IGD_{j,k}^i (i = 1, …, n ;j = 1, …, 30; k = 1, …, m) stands for the IGD value w.r.t. the ith component task obtained in the jth run at the kth predefined number of function evaluations. Note: n=2 and m=100 for 2-task benchmark problems, while n=50 and m=1000 for 50-task benchmark problems.

The first column stores the predefined numbers of function evaluations at which intermediate results are recorded. The subsequent columns store intermediate results for each of 30 runs with each run occupying n consecutive columns w.r.t. n component tasks, respectively. Note: The comma is used as a delimiter to separate any two numbers next to each other in a row. As an example, “.txt” files obtained by MFEA are provided as reference.

(3) Overall ranking criterion
To derive the overall ranking for each algorithm participating in the competition, we will take into account of the performance of an algorithm on each component task in each benchmark problem under varying computational budgets from small to large. Specifically, we will treat each component task in each benchmark problem as one individual task, ending up with a total of 518 individual tasks. For each algorithm compared for ranking, the median IGD value over 30 runs will be calculated at each checkpoint corresponding to different computational budgets for each of 518 individual tasks. Based on these calculated data, the overall ranking criterion will be defined. To avoid deliberate calibration of the algorithm to cater for the overall ranking criterion, we will release the formulation of the overall ranking criterion after the competition submission deadline.

SUBMISSION GUIDELINE
please archive the following files into a single .zip file and then send it to mtocompetition2024@gmail.com 2024mtocompetition@gmail.com before the competition submission deadline (30 June 2024):

For participants in MTSOO: 19 ".txt" files (i.e., "MTOSOO_P1.txt", … , "MTOSOO_P10.txt"), "param_SO.txt" and "code.zip".

For participants in MTMOO: 19 “.txt” files (i.e., "MTOMOO_P1.txt", … , "MTOMOO_P9.txt"), "param_MO.txt" and "code.zip".

For participants in both MTSOO and MTMOO: 38 ".txt" files (i.e., the required ".txt" files for both MTSOO and MTMOO), "param_SO.txt", "param_MO.txt" and "code.zip".

Here, "param_SO.txt" and "param_MO.txt" contain the parameter setting of the algorithm for MTSOO and MTMOO test suites, respectively. "code.zip" contains the source code of the algorithm which should allow the generation of reproducible results.

If you would like to participate in the competition, please kindly inform us about your interest via email (mtocompetition2024@gmail.com) so that we can update you about any bug fixings and/or the extension of the deadline.

COMPETITION ORGNIZERS:

Yaqing Hou
School of Computer Science and Technology, Dalian University of Technology, China
E-mail: houyq@dlut.edu.cn
Short Bio: Yaqing Hou received the Ph.D. degree in artificial intelligence from Interdisciplinary Graduate School, Nanyang Technological University, Singapore, in 2017. He was a Postdoctoral Research Fellow with Data Science and Artificial Intelligence Research Centre, Nanyang Technological University. He is currently an Associate Professor with the School of Computer Science and Technology, Dalian University of Technology, Dalian, China. His research interests include computational and artificial intelligence, multi-agent learning systems, evolutionary multi-tasking and transfer optimization. He is now an Associate Editor of the IEEE Transactions on Cognitive and Developmental Systems, Memetic Computing. He had co-organized multiple Special Issues on Memetic Computing journal and Applied Soft computing.

Liang Feng
Chongqing University, College of Computer Science, China
E-mail: liangf@cqu.edu.cn
Short Bio:Liang Feng received his Ph.D degree from the School of Computer Engineering, Nanyang Technological University, Singapore, in 2014. He was a Postdoctoral Research Fellow at the Computational Intelligence Graduate Lab, Nanyang Technological University, Singapore. He is currently a Professor at the College of Computer Science, Chongqing University, China. His research interests include Computational and Artificial Intelligence, Memetic Computing, Big Data Optimization and Learning, as well as Transfer Learning. He has been honored with the 2019 IEEE Transactions on Evolutionary Computation Outstanding Paper Award, the 2023 IEEE TETCI Outstanding Paper Award, and the 2024 IEEE CIM Outstanding Paper Award. He is an Associate Editor of the IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE, IEEE CIM, and Memetic Computing. He is also the founding Chair of the IEEE CIS Intelligent Systems Applications Technical Committee Task Force on “Transfer Learning & Transfer Optimization” and the PC member of the IEEE Task Force on “Memetic Computing”. He had co-organized and chaired the Special Session on “Memetic Computing” and “Evolutionary Transfer Learning and Transfer Optimisation” held at IEEE CEC since 2016.

Kai Qin
Department of Computer Science and Software Engineering
Swinburne University of Technology, Australia
E-mail: kqin@swin.edu.au
Website: http://www.alexkaiqin.org/
Short Bio:Kai Qin is a Professor at Swinburne University of Technology, Melbourne, Australia. Currently, he is the Director of Swinburne Intelligent Data Analytics Lab and the Deputy Director of Swinburne Space Technology and Industry Institute. Before joining Swinburne, he worked at Nanyang Technological University (Singapore), the University of Waterloo (Canada), INRIA Grenoble Rhône-Alpes (France), and RMIT University (Australia). His major research interests include machine learning, evolutionary computation, collaborative learning and optimization, computer vision, remote sensing, services computing, and edge computing. He was a recipient of the 2012 IEEE Transactions on Evolutionary Computation Outstanding Paper Award and the 2022 IEEE Transactions on Neural Networks and Learning Systems Outstanding Associate Editor. He is currently the Chair of the IEEE Computational Intelligence Society (CIS) Student Activities and Young Professionals Sub-committee, the Vice-Chair of the IEEE CIS Neural Networks Technical Committee, the Vice-Chair of the IEEE CIS Emergent Technologies Task Force on “Multitask Learning and Multitask Optimization”, the Vice-Chair of the IEEE CIS Neural Networks Task Force on “Deep Edge Intelligence, and the Chair of the IEEE CIS Neural Networks Task Force on “Deep Vision in Space”. He serves as the Associate Editor for several top-tier journals, e.g., IEEE TEVC, IEEE TNNLS, IEEE CIM, NNs, and SWEVO. He was the General Co-Chair of the 2022 IEEE International Joint Conference on Neural Networks (IJCNN 2022) held in Padua, Italy, and was the Chair of the IEEE CIS Neural Networks Technical Committee during the 2021-2022 term.

Abhishek Gupta
Singapore Institute of Manufacturing Technology (SIMTech), Agency for Science, Technology and Research (A*STAR), Singapore
E-mail: abhishek_gupta@simtech.a-star.edu.sg
Short Bio: Abhishek Gupta received the Ph.D degree in Engineering Science from the University of Auckland, New Zealand, in 2014. Over the past 5 years, Dr. Gupta has been working in the area of Memetic Computation, with particular focus on developing novel theories and algorithms in the topics of evolutionary transfer and multitask optimization. His pioneering work on evolutionary multitasking, in particular, was bestowed the 2019 IEEE Transactions on Evolutionary Computation Outstanding Paper Award by the IEEE Computational Intelligence Society (CIS). He is Associate Editor of the IEEE Transactions on Emerging Topics in Computational Intelligence, and is also the founding Chair of the IEEE CIS Emergent Technology Technical Committee (ETTC) Task Force on Multitask Learning and Multitask Optimization. He is currently appointed as a Scientist in the Singapore Institute of Manufacturing Technology (SIMTech), Agency for Science, Technology and Research (A*STAR). He also jointly serves as an Adjunct Research Scientist in the Data Science and Artificial Intelligence Research Center, School of Computer Science and Engineering, Nanyang Technological University, Singapore.

Yuan Yuan
Department of Computer Science and Engineering, Michigan State University, USA
E-mail: yyuan@msu.edu
Short Bio:Yuan Yuan is a Postdoctoral Fellow in the Department of Computer Science and Engineering, Michigan State University, USA. He received the PhD degree with the Department of Computer Science and Technology, Tsinghua University, China, in July 2015. From January 2014 to January 2015 he was a visiting PhD student with the Centre of Excellence for Research in Computational Intelligence and Applications, University of Birmingham, UK. He worked as a Research Fellow at the School of Computer Science and Engineering, Nangyang Technological University, Singapore, from October 2015 to November 2016. His current research interests include multi-objective optimization, genetic improvement, and evolutionary multitasking. Two of his conference papers were nominated for the best paper at the GECCO 2014 and GECCO 2015, respectively.

Eric Scott
Department of Computer Science, George Mason University, USA
E-mail: escott8@gmu.edu
Short Bio: Eric Scott is a PhD candidate at George Mason University and a Senior Artificial Intelligence Engineer at MITRE Corporation in Northern Virginia. His research focuses on heuristic optimization algorithms and their applications to simulation and modeling in a variety of fields. He holds a double B.Sc. in Computer Science and Mathematics from Andrews University in Berrien Springs, Michigan, and a M.Sc. in Computer Science from George Mason University.

Yew-Soon Ong
School of Computer Science and Engineering, Nanyang Technological University, Singapore
E-mail: asysong@ntu.edu.sg
Website: http://www.ntu.edu.sg/home/asysong/
Short Bio:Yew-Soon Ong is Professor and Chair of the School of Computer Science and Engineering, Nanyang Technological University, Singapore. He is Director of the A*Star SIMTECH-NTU Joint Lab on Complex Systems and Programme Principal Investigator of the Data Analytics & Complex System Programme in the Rolls-Royce@NTU Corporate Lab. He was Director of the Centre for Computational Intelligence or Computational Intelligence Laboratory from 2008-2015. He received his Bachelors and Masters degrees in Electrical and Electronics Engineering from Nanyang Technological University and subsequently his PhD from University of Southampton, UK. He is founding Editor-In-Chief of the IEEE Transactions on Emerging Topics in Computational Intelligence, founding Technical Editor-In-Chief of Memetic Computing Journal (Springer), Associate Editor of IEEE Computational Intelligence Magazine, IEEE Transactions on Evolutionary Computation, IEEE Transactions on Neural Network & Learning Systems, IEEE Transactions on Cybernetics, IEEE Transactions on Big Data, International Journal of Systems Science, Soft Computing Journal, and chief editor of Book Series on Studies in Adaptation, Learning, and Optimization as well as Proceedings in Adaptation, Learning, and Optimization He is also guest editors of IEEE Transactions on Evolutionary Computation, IEEE Trans SMC-B, Soft Computing Journal, Journal of Genetic Programming and Evolvable Machines, co-edited several books, including Multi-Objective Memetic Algorithms, Evolutionary Computation in Dynamic and Uncertain Environments, and a volume on Advances in Natural Computation published by Springer Verlag. He served as Chair of the IEEE Computational Intelligence Society Emergent Technology Technical Committee (ETTC) from 2011-2012, and has been founding chair of the Task Force on Memetic Computing in ETTC since 2006 as well as a member of IEEE CIS Evolutionary Computation Technical Committee from 2008 - 2010. He was also Chair of the IEEE Computational Intelligence Society Intelligent Systems Applications Technical Committee (ISATC) from 2013-2014. His current research interests include computational intelligence spanning memetic computing, evolutionary optimization using approximation/surrogate/meta-models, complex design optimization, intelligent agents in game, and Big Data Analytics. His research grants comprises of external funding from both national and international partners that include National Grid Office, A*STAR, Singapore Technologies Dynamics, Boeing Research & Development (USA), Rolls-Royce (UK) and Honda Research Institute Europe (Germany), National Research Foundation and MDAGAMBIT. His research work on Memetic Algorithm was featured by Thomson Scientific's Essential Science Indicators as one of the most cited emerging area of research in August 2007. Recently, he was selected as a 2015 Thomson Reuters Highly Cited Researcher and 2015 World's Most Influential Scientific Minds. He also received the 2015 IEEE Computational Intelligence Magazine Outstanding Paper Award and the 2012 IEEE Transactions on Evolutionary Computation Outstanding Paper Award for his work pertaining to Memetic Computation.

REFERENCES:
[1] R. Caruana, “Multitask learning”, Machine Learning, 28(1): 41-75, 1997.
[2] K. Swersky, J. Snoek and R. P. Adams, “Multi-task Bayesian optimization”, Proceedings of the 26th International Conference on Neural Information Processing Systems (NIPS'13), pp. 2004-2012, Lake Tahoe, Nevada, USA, December 5-10, 2013.
[3] A. Gupta, Y. S. Ong and L. Feng, “Multifactorial evolution: Toward evolutionary multitasking”, IEEE Transactions on Evolutionary Computation, 20(3):343-357, 2016.
[4] Y. S. Ong and A. Gupta, “Evolutionary multitasking: A computer science view of cognitive multitasking”, Cognitive Computation, 8(2): 125-142, 2016.
[5] A. Gupta, Y. S. Ong, L. Feng and K. C. Tan, “Multi-objective multifactorial optimization in evolutionary multitasking”, accepted by IEEE Transactions on Cybernetics, 2016.
[6] K. K. Bali, A. Gupta, Y. S. Ong, and P. S. Tan. "Cognizant Multitasking in Multi-Objective Multifactorial Evolution: MO-MFEA-II." IEEE Transactions on Cybernetics, 2020.
[7] Gupta, A., Zhou, L., Ong, Y. S., Chen, Z., & Hou, Y. "Half a dozen real-world applications of evolutionary multitasking, and more." IEEE Computational Intelligence Magazine 17.2 (2022): 49-66.
[8] B. S. Da, Y. S. Ong, L. Feng, A. K. Qin, A. Gupta, Z. X. Zhu, C. K. Ting, K. Tang and X. Yao, “Evolutionary multitasking for single-objective continuous optimization: Benchmark problems, performance metrics and baseline results”, Technical Report, Nanyang Technological University, 2016.
[9] Y. Yuan, Y. S. Ong, L. Feng, A. K. Qin, A. Gupta, B. S. Da, Q. F. Zhang, K. C. Tan, Y. C. Jin and H. Ishibuchi, “Evolutionary multitasking for multi-objective continuous optimization: Benchmark problems, performance metrics and baseline results”, Technical Report, Nanyang Technological University, 2016.
[10] P. Czyzzak and A. Jaszkiewicz, “Pareto simulated annealing– a metaheuristic technique for multiple-objective combinatorial optimization”, Journal of Multi-Criteria Decision Analysis, 7:34-47, 1998.