Office of Learning and Information Technology
UWS CIO Council, September 21, 2006
UWS CIO Council Retreat and Meeting
September 21, 2006
DoIT platform management & procedures for outages, emergencies, etc.
Common Systems & UW HR Executive Committee briefing
UW-Milwaukee’s email and calendaring project
- Pandemic planning
- Strategic Sourcing
UW System Records Management report
CIOs and their Representatives
Kathy Pletcher congratulated the Learn@UW staff on a successful fall startup. Cora Marrett also expressed her congratulations to Ed Meachen. Alan Foley received no calls concerning performance issues from site administrators or the CIOs this fall, compared to frequent calls during the same period last year.
The Learn@UW Utility was charged by the UW System CIO Council in fall 2005 to provide a set of recommendations for improvements prior to fall 2006 to ensure that student and faculty would not experience performance problems. This year, Learn@UW was able to successfully prevent fall startup performance issues and provide sufficient server capacity to enable continued growth.
Learn@UW deployed new database hardware in a high availability cluster on July 25. They upgraded to Tier 1 disk storage for the database servers, which provides better performance and reliability. In response to a recommendation from UW–Milwaukee, Learn@UW moved to the 64 bit Windows Operating System, even though D2L doesn't support it officially, for better performance and to be in position for migration to SQL2005 . The 32 bit architecture is still running on the application servers and file servers. Learn@UW also conducted load tests on the production configuration (during non-peak hours) using Empirix e-Load with a 30 day lease of 500 virtual users as opposed to the normal test of 100 virtual users. The Learn@UW test results matched those from D2L.
One new application server, with dual-core CPUs, was deployed for the start of fall. The application servers are at the top edge of their performance and will max out at about 200 ASP requests per second. Performance models show that this would only put a CPU load of 26% on the database server. Discussions have begun with with Microsoft and D2L to find a scalable configuration for the application servers.
To date, Learn@UW has spent approximately 80% of the budget that was projected in February, not including Learn@UW staff time. Only a one year software assurance contract was purchased from IBM since there was no price break for a longer agreement.
During the first 10 days of the semester, there was a 22% increase in the number of login attempts versus the same period last year.* There was also a 110% decrease in the number of errors. On Monday, September 11, there was a record high of 144,725 logins per day.
Number of login attempts
Sept 5-12, 2005
Sept. 6-13, 2006
* The number of login attempts was
inflated during the start of the semester in fall 2005 due to
performance issues and timeouts. The report for 2006 represents valid login attempts
under stable performance, so the actual growth is probably larger.
This year, there has been a shift from institutional reporting to automated Learn@UW reporting of enrollments, students and active courses at each institution. A couple UWS institutions auto-generate D2L shells for all courses on campus, which must be kept in mind when comparing their counts to other institutions. Similarly, UW–Madison is auto-enrolling all students, but the students are manually activated. Perhaps a finer grained analysis of the type of usage by faculty and students will be available through new reporting tools in version 8.0.
The next steps are to:
- Complete the Capacity Planning Review, make any necessary adjustments to the model, and plan for spring and fall 2007 anticipated growth.
- Continue to encourage the D2L company to support SQL Server 2005 and encompass 64 bit architecture across the full environment.
- Engage Microsoft, D2L and hardware vendors to determine the best solution to scale up the application servers.
- Review reports from the UW System institutions for anticipated growth in the number of courses and enrollments for 2007 and beyond to ensure the hardware platform can meet the expected growth.
- Continue to work with the MultiState users group, in particular with MNSCU, to share solutions for the issues that occur in large-scale implementations of D2L.
The Identification, Authentication and Authorization project (IAA) is an identity management system that links institutions across the UWS. It provides:
- identity management
- user and account management
IAA reduces complexity, cost and processing for Common Systems, such as Learn@UW. It creates records from campus student systems, IADS and other data sources. It allows students and faculty to use their local usernames and passwords when using systemwide services. It does not however store user passwords. It brokers authentication, it doesn't replace local authentication.
The UWS sponsors the project, which has partners at the UWS institutions who provide functional and technical support, e.g., data feeds and help with local policy issues. The IAA Governance Working Group provides stewardship and oversees the appropriate use. The DoIT Middleware System Technology group provides development, operations and support. The UWS MILER group assists institutions with local support and integration. FASTAR provides infrastructure operations and support.
A fundamental operating principle is that IAA supports local institutional policies and identity management mechanisms. Data ownership and policy decisions stay at the institutions.
Currently, Learn@UW does not receive information from IAA as to whether an individual is a student or instructor. That information comes from course rosters. Multiple signons are still necessary for students who take classes at more than one institution because there is not a systemwide login for each Common System system. IAA does create its own systemwide publicly visible identifier (SPVI) for each person, which could possibly be leveraged to provide systemwide login functionality in the future if more applications would use it.
IAA is currently used by, or underway, for:
- Learn@UW (authentication)
- MINDS@UW (authentication and authorization)
- UW Libraries Ex Libris MetaLIB federated searching (authentication pilot at UW–Madison)
- UW Libraries (authentication to replace EZ-Proxy)
- SFS (authorization)
- SFS WISDM (authorization)
- Wisc online software catalog (authentication)
- HR to Student Information Interface at UW–Milwaukee
- Online web surveys primarily at UW–Madison (authentication)
- ECRT Effort Reporting for time reporting on grant activities at UW Extension, UW–Madison and UW–Milwaukee
Integrating KRONOS into IAA is currently in the hands of the vendor to address technology and data problems. The intent is that all UW System libraries will move to IAA. Bringing authentication services to new applications is a fairly standard process. The preferred mechanism, which is not used at Learn@UW, is for the user to choose their institution from a list and enter their local user ID and password.
The efforts of the IAA Governance Working Group are key to getting new applications in place. The governance group has generated:
- Guidelines for use
- Memorandum of Understanding for the UWS institutions with respect to appropriate use of the data
- Process to request use
- Process to request additional data elements (over 20 requests have been reviewed)
- Public relations efforts, including reports, presentations and liaison functions
Ron Kraemer, Joanne Berg and Carrie Regenstein will be doing a presentation on IAA governance at EDUCAUSE 2006. Budgeting for IAA is based on the needs of the underlying infrastructure. Additional projects need to bring their own funding. UW–Madison also provides significant support for the IAA effort.
The next steps for IAA include:
- additional application integrations
- exploring Service Oriented Architecture (SOA) for web services and data exchange
- exploring the use of federated authentication and authorization technologies, such as Shibboleth.
Shibboleth is an evolving higher education mechanism for accomplishing these same types of functions across institutions and realms.
The current capabilities for backup and recovery at UW–Madison include:
- "Bucky Backup" storage management for all systems to disk (fast recovery) and tape (older versions)
- OS images for all systems stored on disk
- Data mirroring for all systems
- Divergent power at the two data centers: (warm and tape backup site) and (primary site)
- Incident managers trained in disaster procedures
Disaster classifications are:
- Complete loss of a data center infrastructure and the equipment in it, e.g., a flash flood
- Complete loss of a data center infrastructure, e.g., losing a main building transformer
- Temporary loss of a data center infrastructure, e.g., a power failure
Current recovery status for these three categories is:
- OS images, data mirror and backup, priority list, recovery procedures, training of incident managers, recovery of critical systems within several weeks of an incident
- Secondary cold site on stand by, complete data mirror and backup, recovery of critical systems within days of the incident
- Procedures, documentation, recovery of all systems within hours of incident
In the case of a category 1 incident, if no hardware could be re-used, the longest time lapse would be obtaining new hardware.
The data center is actually two separate rooms. In a recent 6" rainstorm, water accumulated in a depression between buildings and began to accumulate in the smaller of two rooms. Normally shutting down email would be the last resort. Because of the way the disaster occurred, it was necessary to shut down the smaller room first, which included email and the communications suite. As water begin to flow under the raised floor supporting the storage systems, the decision was made to shut down all of the equipment to prevent it from shorting out. After the data center was dried out, the planned procedures were followed to power up the systems in order by the appropriate responsible parties.
The cost of completely replicating the data center would be millions of
dollars. Backup power for the entire data center is the highest priority, but
that also requires a multi million dollar piece of equipment that needs its own
building. Discussions are taking place on the UW–Madison campus regarding
continuity in general. Meanwhile, the State of
Ed Meachen suggested that any large scale plan for UW–Madison be a Board of Regents effort that includes the entire UW System because of the need to keep Common Systems running.
David Lois asked how the UWS institutions can stay informed in the midst of an incident. Because of the early impact on email, word of the summer rainfall spread by telephone. Perhaps there should be an emergency cell phone list for the CIOs. There is also a systemwide continuity of operations project underway.
The BadgerNet Converged Network (BCN) did not have any significant issues during the start of classes. There are still minor packet losses at two UWS institutions. A ticket is open and weekly reports on the status are being received from the vendors. The severity of the losses appear to be correlated with utilization.
The UW–Superior connection had problems with the initial strands of fiber. There was a fallback to the legacy circuit for a day followed by re-establishment of the BCN connectivity. It is not clear why the fiber problems were not found in the initial testing. WiscNet has test servers at each of the UWS institutions to track performance. Some state agencies are now also becoming concerned about the black box nature of the network.
The WiscNet strategic plan focuses on maintaining the quality of the network and facilitating participation by members of the WiscNet community. New workgroups are being formed around topical issues and services. The workgroups set their own governance and direction and recommend changes and services to WiscNet. A workgroup is being formed for Advanced Networks. Chip Eckardt and Jeff Bartig will be the co-chairs. The first meeting will be November 16. Both technical and managerial representatives from the UWS institutions are welcome. The group will likely meet four times during the first year at various locations around the state.
Documents were recently emailed to all principle administrative groups in the UW System regarding the pre-implementation timeline for a new HR system, which was endorsed by the HRIS Executive Committee. It was decided to separate the time periods for scoping HR from that of supply chain/supplier relationship for staffing and budgetary reasons.
Tomorrow, Debbie Durcan will present the Common
Systems budget for next year to the chancellors. Ed Meachen has been invited to be member of the executive
committee for the State of
A Shared Financials Executive Advisory Committee has been recently constituted. A similar committee may be needed for Service Oriented Architecture (SOA) and middleware.
The Common Systems Review Group will meet next week.
About 3 years ago, UWM began a series of Core Services initiatives that distribute the planning and decision making for new projects to the broad campus community. The goal is build services that are so good that people will want to adopt them. There are as many decentralized technology workers on the UWM campus, as in the computer center. The deans would like these individuals to work on applied functions in their disciplines, not merely replicating services. Ultimately, it will be the decision of each academic unit as to whether or not it adopts a Core Service.
There have been several successful Core Service projects, including a
standard Windows desktop ordering process that occurs on a quarterly basis for a
limited number of configurations. That project was led by an Assistant Dean in
For a decentralized research institution like UWM to settle on one email and calendaring package would be phenomenal. Denise Babin from the UWM Libraries led the first team, which identified the need for including calendaring in addition to email. The second team took on the charge of selecting an integrated package under the leadership of Professor Jacques Du Plessis.
A new website is communicating the process to the campus (emailfuture.uwm.edu). This morning, there
was a focus group meeting with current Groupwise
users. A meeting with the
The difficulty of having all team members at a series of meetings was addressed by collaborating via on online environment. A site visit to UW–Parkside revealed the challenges of bringing too many vendors to campus. UWM's CMS selection process showed the downside of bringing different vendors to campus on different days. The team focused primarily on user needs and total cost of ownership (TCO). A primary consideration was a choosing a product that would run on all common platforms and interface with Outlook.
There are five excellent alternatives among the finalists and all would work equally well at different universities. Significant effort was expended to look at TCO for an institution the size of UWM. For example, it appears that Exchange scales well for institutions in the range of 5,000 - 10,000, but not as well for UWM's 60,000 mailboxes. Two Microsoft Exchange VARs responded to the RFP and were within a few points of each other in the evaluation but different from the other vendors in regards to TCO. Exchange would likely require an initial $2.5M investment and re-engineering of the university's email systems and processes. Therefore, it is important for the campus to understand that the equanimity of the evaluation process.
Ron Kraemer inquired about the embedding of current email/calendar systems
within the work processes of academic departments. Bruce Maas replied that these
issues are likely to arise as UWM goes forward. There will be no compulsion for
anyone to convert. They will likely need to weigh the advantages of local
workflow versus enterprise workflow. This will likely only be an issue for the
There are discussions around state government and the UWS pertaining to emergency planning and restoration 30 days after a crisis. Health officials are saying that there will be a pandemic flu at some point in the future. The UWS CBOs are appointing people to work with the UW–Madison police for planning purposes. Ruth Anderson gave the committee a heads up regarding an email she will send regarding the feasibility of entire staffs working from home. Ron Kraemer pointed out that IT staff would likely be ill at the same rate as other staff. Chip Eckardt pointed out that computer lab software is often not licensed for home use. Elena Pokot noted that most D2L courses are still meeting face to face 50% of the time.
There was a recent meeting with the new Microsoft sales rep. He will likely have a greater focus on higher ed. Microsoft has various support levels and contracts beyond the basic Software Assurance. The UWS has four Essentials contracts for putting issues into Microsoft's tracking system. Perhaps a Premier level of support for the UW System would be of greater value for much the same cost. On the other hand, Microsoft is changing their support products and may be moving toward a per-hour basis. UW-Eau Claire has used the various support products in the past and questions whether there is significant extra value in Premier support. If Dell, D2L and the UW System would have the same level of support, Microsoft says it would be able to provide a greater level of troubleshooting.
Microsoft purchased Defender, which produces the Forefront anti-virus product. With the Vista OS, it will be possible to buy CALs for Forefront at a few dollars per copy. The position of UW–Madison is that Forefront should be included at no cost. Since Forefront doesn't work for Macs, there is still a need for an anti-virus product for Macs on campuses, such as UW–Madison, that are one quarter Macs.
Microsoft is also looking to offer enterprise level CALs, which may or may not be required for certain features of products.
ORACLE has suggested that the UWS acquire a Priority Service level of support, but no pricing has yet been offered. UW–Parkside is considering using the UPK toolkit for training. The UWS already has rights to use it and ORACLE is willing to provide web or regional onsite demonstrations. It might be valuable for user training for the new HRIS system.
The state's e-procurement effort is a much broader than just a software module. It is a widespread initiative of supply chain management. Using the term e-procurement is inaccurate and misleading. A better acronym or label is needed.
Berbee Information systems has been bought by CDW. It is not yet clear how if this will affect their interactions with the UWS.
DOA will likely continue its strategic sourcing efforts, but they are currently working on modifications to existing contracts. A prime vendor for books for the UWS Libraries is being sought.
Within the Grainger contract, there is an exemption for buying items under $100 locally, e.g., at local hardware stores.
Both UW–Madison and UW–Milwaukee already have Xythos licenses, which complicates the issue of possibly
licensing the remainder of the UWS. New terms were offered by Xythos yesterday which include a price reduction and obviate
the need to count the number of users of
The number of additional UWS institutions that have expressed interest in Xythos to date are insufficient to swing the deal. UW–Madison in particular is interested in additional licenses. Potential financial scenarios will be modeled based upon participation by different numbers of institutions.
Xythos has agreed to visit other potential customers in the next few weeks and set up eval sites within a few hours.
Lorie Voss has said that the rest of the UW System cannot buy a Content Management System (CMS) under the UWM CommonSpot contract. The options are to do either a CMS RFP or a bid. Lorie Docken will check with Lori Voss regarding strategy and the workload in her office.
The UWS has a new account representative. The goal is move to a CPU-based license to obviate the counting of users. It will probably take a couple months to work this out. In the meantime, FASTAR will remove the cap on the adding of Hyperion users. FASTAR will also contact each campus to verify the licenses that they are using to see if all are necessary. The CIOs are interested in simply receiving a list of the users at their institutions.
The Board of Regents is concerned about the security incidents that have been reported at other institutions in the Chronicle of Higher Education and elsewhere. Therefore, fieldwork for a broad UWS security review is underway. It is not a full security audit from a technical perspective. Instead, the interest is in the structure and management of security functions at the institutions as well as policies, authorities, responsibilities and practices. Eight institutions, including UW Colleges and UW Extension, are being visited. Others will be contacted by phone over the next couple months.
Ed Melchior, the Information Systems Auditor at UW–Milwaukee, has a background in both higher education and industry. He explained that IT audits are similar to, and complementary with, financial audits. The goal of an IT audit is information assurance which is about data being available, but not where it isn't supposed to be. At UWM, he also does business process consulting, often in partnership with the IT division.
The components of IT audits are:
- identifying risks
- identifying controls
- standards and policies
- segregations of duties
The common mission of IT and Audit is focusing on students. Auditors try to set a tone of helping their clients make the business better. Synergies are available between central IT and Audit. Given adequate resources, most units wouldn't need the assistance of auditors. However, resources tend to be limited and auditors can help with specific expertise. They can also help IT make the case for bigger budgets and to validate current practices.
Early involvement of auditors in planning and system implementation eases the burden later on by establishing the right processes and procedures from the beginning. Auditors are also a valuable resource in establishing and validating the need for policy.
At UWM, Internal Audit works closely with the IT Security Office and helps with enforcement investigation. Their combined goal is to establish a culture of security on campus.
UW-Milwaukee is implementing the NeXpose Vulnerability Management toolset. It includes:
- host reviews
- database security, e.g., ORACLE databases are shipped with include known default passwords enabled
- application security
- web server security
- network security
- user policy
UWM's pilot of one Windows box, one Unix box and one Cisco box took only 30 minutes of clock time. It can scan for:
- HIPAA compliance
- Payment Card Industry certification
- DMZ security
- penetration weaknesses
- DNS vulnerabilities
As an auditor, Ed doesn't want to have system passwords to do his job. NeXpose provides a credentialing system for the system administrator to encrypt their login and password and only uses the credentials to log in to the systems to be scanned. NeXpose's generates reports that are ready to give to stakeholders. They include:
- executive summary (two pages with no technical jargon)
- baseline comparison (changes from the previous audit)
- remediation plans with detailed instructions for system administrators
- policy evaluations, e.g., three password attempts before lockout
- comprehensive report card
- audit summary
Ed Melchior is eager to help other UWS institutions run this product because the licensing terms are very generous. Scans at other institutions can be done over the network from UWM.
The UWS Records Management Working Group is putting together a document for the Board of Regents. The goals are to create a policy for records management under the authority of Chapter 36 of the Wisconsin Statutes. The works of UWS employees are by default public records. The Regents can provide systemwide guidance and resources to address this responsibility. One study showed that some 98% of today's records are electronic.
Formal feedback for the draft policy will be requested through the records managers at each institution. The question at hand is how the IT professionals within the UWS can help to guide the formation and implementation of the policy over the next couple months. Kathy Pletcher will help keep the lines of communication open.
Ron Kraemer suggested that the role of IT with respect to records is similar to that of the Library with respect to books, i.e., a repository. Laura Dunek replied that IT is actually more involved in the creation of records than the library is with the creation of books. IT also creates records through system logs which enable inferences about peoples' behavior. There are business choices made regarding what is logged, either because of explicit guidance or the lack of it.
Ed Meachen noted the need for the primary involvement of chief academic officers and CBOs. The CIOs need to define what the UWS systems do and make proposals to the operational officers.
Judy Caruso noted that DOA has suggested a records retention schedule which would likely be overkill for the UWS.
The October CIO meeting will be held after the ITMC meeting in
Meeting dates, Directory of UW CIOs, Meeting Summaries: www.uwsa.edu/olit/cio/