Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP Path: utzoo!mnetor!uunet!seismo!mcvax!philmds!prle!nvpna1!knehans From: knehans@nvpna1.UUCP (Frits Knehans 43250) Newsgroups: comp.os.vms Subject: 8800 as a BATCH-machine ? Message-ID: <245@nvpna1.UUCP> Date: Mon, 24-Aug-87 11:31:17 EDT Article-I.D.: nvpna1.245 Posted: Mon Aug 24 11:31:17 1987 Date-Received: Tue, 25-Aug-87 01:59:53 EDT Reply-To: knehans@nvpna1.UUCP (Frits Knehans 43250) Distribution: world Organization: Philips Research Labs, Eindhoven Lines: 246 Having never set up a VAX as a BATCH-machine we would like to get some comments, suggestions on how you did it. Or some comments on how *not* to do it. The machine is a 128Mb 8800 VAX/VMS, which is a cluster-member. The environment is a research-environment, using a lot of different applications, each of them with their own specific resource-needs. ********* Here is the rough idea of how we want to set it up. (NOTE: not all figures are correct, so don't *flame*) The queue set up as defined below gives a maximum of 36 concurrent jobs at various priorities and memory allocations. If all the queues were used to their maximum extent 24 hours per day, this would represent some 200K pages of memory being constantly in use (please check the figure...!). Thus some of the restrictions etc may be unnecessary (comments please). However this figure does not take into account the system queues (archiver, hyper, backup) or the plotter batch queues. Names of batch queues have not been decided, and all are represented by a number (eg 6$batch). Some of the well established queues are still available - eg FAST, SYS$BAT30, but their characteristics have been modified. 1 Description of Available Batch Queues General Batch queues Except for specialised job processing, the general batch queue environment is the place to start for most users. This section deals with these queues and points out some performance issues and also gives some general hints on queue usage. The queues are restricted in terms of CPU time restrictions, memory allocation, concurrent job processing and priority. Users are encouraged to develop their skills within the environment, and there are some 'test' queues available. Some hints on how to analyze individual job parameters is also given. The ultimate aim is to establish a suite of batch queues which provide the most efficient use of the available resources. 1.0 Batch Queue Name: 1$batch Queue Characteristics BASE_PRORITY=5 JOB_LIMIT=8 CPUDEFAULT=00:10 CPUMAXIMUM=00:10 WSDEFAULT=500 WSEXTENT=5000 Suggested Queue Usage The limiting factor on this queue is its CPUMAXIMUM rating of 10 minutes. This queue is therefore suited to test runs on compiles, DCL command procedures, local area network copying and any small routines. The job limit is eight concurrent jobs. The memory allocation is generous, the base priority higher than interactive and other batch job processing. 2.0 Batch Queue Name: 2$batch Queue Characteristics BASE_PRORITY=4 JOB_LIMIT=6 CPUDEFAULT=00:30 CPUMAXIMUM=00:30 WSEXTENT=5000 Suggested Queue Usage Effectively the next stage in the general purpose batch processing environment, this queue offers a CPUMAXIMUM of 30 minutes, with an interactive base priority. Used for longer command procedures, medium sized routines etc. Memory usage is restricted to 5000 pages maximum in the working set. 3.0 Batch Queue Name: 3$batch Queue Characteristics BASE_PRORITY=4 JOB_LIMIT=6 CPUDEFAULT=01:00 CPUMAXIMUM=01:00 WSDEFAULT=500 WSEXTENT=2000 Suggested Queue Usage A generalised all purpose batch queue. Notice that the CPUMAXIMUM is set for one hour. There is a relatively restricted access to memory (WSEXTENT=2000) but base priority is equivalent to interactive processing. This queue should be used as the next step up from the 'FAST' and 'INTERMEDIATE' queues, for longer processing, for example several compiles in a command procedure, with various resulting reports and or printouts/plots etc. 4.0 Batch Queue Name: 4$batch Queue Characteristics BASE_PRORITY=4 JOB_LIMIT=4 CPUDEFAULT=08:00 CPUMAXIMUM=08:00 WSDEFAULT=500 WSEXTENT=2000 Suggested Queue Usage This queue is appropriate for background batch processing, on a (2)daily basis. This queue represents the upper limit of the general batch processing environment. For jobs which require more CPU time, the specialised queues should be used. SPECIALISED BATCH QUEUES The memory limitations set in the generalised queues are substantially changed in the specialised queues, as are the restrictions on CPU time. In these cases however, base priority drops as results will not generally be achieved within an eight hour working day. Therefore these queues are specifically aimed at long term batch processing for a finished 'product', whereas the general batch queues are designed for intermediate stages within the 'product' development. Remember that jobs which run on all the general queues will run equally as well on the specialised queues, but users who regularly abuse the specialised queues (as they have restricted job number limits) may well incur the wrath of their colleagues if a 2 hour job is run on a 12 hour queue on a regular basis! As the names of the queues suggest, these are aimed at particular image runs, which are known to be large memory and CPU users. 5.0 Batch Queue Name: 5$batch Queue Characteristics BASE_PRORITY=3 JOB_LIMIT=2 CPUDEFAULT=12:00 CPUMAXIMUM=INFINITE WSDEFAULT=1000 WSEXTENT=8000 Suggested Queue Usage This queue represents the testing queue for 'top of the range' jobs. If a user is very unsure about the amount of CPU time a job is going to take (for instance a new procedure, or a relatively inexperienced user) this is the queue to use! The only restriction is the number of concurrent jobs processed on this queue; there is no restriction on CPU time, and memory allowances are generous. However, it should be used to obtain the minimum and maximum system resources required by the job; subsequent job submitting should be made to a more appropriate queue. 6.0 Batch Queue Name: 6$batch Queue Characteristics BASE_PRORITY=2 JOB_LIMIT=4 CPUDEFAULT=INFINITE CPUMAXIMUM=INFINITE WSDEFAULT=1000 WSEXTENT=8000 Suggested Queue Usage As the name of the queue suggests this queue is designed for CMSK procedures. Memory quotas are generous, and there are no restrictions on CPU. Up to four concurrent jobs are allowed, and although the base priority level of 2 will extend elapsed time, there are no CPU restrictions. 7.0 Batch Queue Name: 7$batch Queue Characteristics BASE_PRORITY=1 JOB_LIMIT=2 CPUDEFAULT=INFINITE CPUMAXIMUM=INFINITE WSDEFAULT=2000 WSEXTENT=16000 Suggested Queue Usage This queue should only be used for the CHAMELEON software suite - ie images like LUDIEC, which require large working set sizes. The introductory base priority level of 1 makes this a relatively slow queue, and two concurrent jobs will be processed on this queue. 8.0 Batch Queue name: 8$batch Queue Characteristics BASE_PRORITY=4 JOB_LIMIT=2 CPUDEFAULT=INFINITE CPUMAXIMUM=INFINITE WSDEFAULT=4000 WSEXTENT=16000 Suggested Queue Usage This queue represents the top of the range. A large amount of memory is dedicated to this queue, the only limit being the number of concurrent jobs available for processing. This queue is for final product processing, when an unrestricted environment is desired. 2 Description of Batch Queue Characteristics When deciding on which batch queue to submit a job to, it is useful to judge how much of the systems resources the job is going to use. This will be established initially by trial and error, and checking the log files for the amount of CPU time the job takes and the amount of memory it uses. This can be matched to the available queue characteristics, and any similar jobs can be placed in that queue. 3 Example of Batch Log File Accounting information: Buffered I/O count: 620610 Peak working set size: 1000 Direct I/O count: 1591257 Peak page file size: 23251 Page faults: 1413310 Mounted volumes: 0 Charged CPU time: 0 01:10:16.14 Elapsed time: 0 17:37:17.99 The important pieces of information are the Page fault, Peak working set size and Charged CPU time figures. These can be related to the queue characteristics WSDEFAULT, WSEXTENT and CPUMAXIMUM. The way to relate the logfile information to the queue characteristics is illustrated below. Log file information Queue Characteristics Charged CPU time: 0 01:10:16.14 ~ CPUMAXIMUM Peak working set size: 1000 ~ WSDEFAULT/WSEXTENT Page faults: 1413310 ~ +- 300 second Select a queue with CPUMAXIMUM = INFINITE for this particular job. The peak working set size is 1000 pages, so select a queue with WSDEFAULT equivalent to 1000; if the job requires more memory it will be able to grow to the WSEXTENT of that queue. The page fault rate gives an indication of the efficiency of the memory resource usage. Jobs which pagefault at more than 300 per second should be placed on a queue with larger WS parameters so that memory allocation is decided at the start of the job; this will help the CPU, as heavy pagefaulting uses a large amount of kernel mode CPU, which effectively means that the CPU is doing more work than necessary to run the job. End of rough idea *********** Again, comments, suggestions or if you set it up in an entirely different way please let us know. THANKS IN ADVANCE.