18 Oct 2000, 09:48 UTC
Re: Systems - ERMS
- From
- Beth Perlman beth.perlman@enron.com
- Date
- Wed, 18 Oct 2000 02:48:00 -0700 (PDT)
John,
These issues are being addressed by Steve Stock and Chris Hanz in my area as
well as people on the infrastructure side (in Jenny Rub's organization). We
understand the significance of these problems and they have not gone
unrecognised. There are 3 components to this plan, the DBA fixes are noted
below (I know it's a lot of technical terms, but this is a highly complex
problem). The other 2 pieces are the application and the hardware. I will
include these plans as soon as I receive them.
My team stayed last night with Jeff Gossett and his folks. A number of items
came out of observing what transpires in risk. We feel that there are many
performance enhancements that can be made to the macros to speed up non ERMS
processing. I'll keep you posted on what we find and the plans to fix them.
Beth
Database Observed Problems
? OS - Severe I/O contention (due to 6 years old disk storage device)
? OS - Max CPU utilization (due to wait I/O contention)
? DB - Redo Log Sync (synchronization of Redo information memory with disk)
? DB - Shared Pool fragmentation (due to unbound SQL )
? DB - Inefficient buffer cache utilization (due to inefficient SQL)
Recommendation
Short Term
? Alleviate Shared Pool contention
- increased Shared Pool size
- disabled MTS
- increased session_cached_cursors
Implemented on 10/15/2000
? Alleviate Redo Log Sync
- increased redo_log_buffer
Implemented on 10/15/2000
? Alleviate I/O contention for rollback segment tablespace
- added additional tablespace with 2/3more rollback segments
Implemented on 10/17/2000
? Alleviate I/O contention for redo logs
- reorg disk storage and implement striping for Redo logs
- turn on force direct I/O write for intensive volumes
Under investigation ) 2 days work
? Turn on Veritas tuning feature
- expect 5% improvement
- tested on Shemp today
Required system reboot
? SQL Tuning
Deliver Lists of expensive SQL with recommendation by 10/27/2000
? Reassess CPR migration to different UNIX host
Waiting for disk storage ) 2 to 3 weeks
New 10K located in Ardmore
Long Term
? Latest storage technology
? Capacity planning
? ERMT and CPR application review
? Upgrade ERMT to 8I
John J Lavorato@ENRON
10/17/2000 01:31 PM
To: Beth Perlman/HOU/ECT@ECT, Philippe A Bibi/HOU/ECT@ECT
cc:
Subject: Systems - ERMS
Quoted or forwarded from earlier messages
---------------------- Forwarded by John J Lavorato/Corp/Enron on 10/17/2000 01:25 PM --------------------------- From: Jeffrey C Gossett @ ECT 10/17/2000 09:45 AM To: John J Lavorato/Corp/Enron@Enron cc: Subject: Systems - ERMS John - Again, we are having "big time" systems issues. While we are having the usual length of time issues (a few people were here past midnight last night), the issues that occured last night are much more serious than in the past. The system brought through totally inaccurate information on a few books. (Positions, value, etc.) Luckily, these errors were all caught last night, but if a less experienced person had been on those books, it could have been a disaster. (Traders getting incorrect positions and p&l and interest rate and credit getting incorrect numbers.) I have contacted those in ERMS that need to be involved, but I would appreciate it if you could provide additional support to this effort. Thanks, Jeff