New Release Testing
Definitions
New release testing is the process TCs use to test
an upcoming Verizon systems release that impacts the interface and
business rules between TCs and Verizon. This testing will take place
after Verizon has completed its internal testing of the release.
New release testing is intended for those TCs
which are currently in production with Verizon, submitting and receiving
pre-order or order transactions through an application-to-application
interface (i.e., EDI, CORBA). This process does not apply to the Web GUI
interface.
Quality
Baseline Validation Test Decks and Test Accounts
Verizon has created and will maintain standard
Quality Baseline Validation Test Decks of pre-order and order
transactions that will be used to test a new release. The Quality
Baseline Validation Test Deck is also referred to as the Regression Test
Deck. Verizon will distribute the updated regression test decks for the
upcoming new release through BA Change Control two weeks prior to the
start of TC testing. The regression test decks are posted to the Verizon
Wholesale Web site shortly after their distribution through BA Change
Control.
Verizon will run the regression test decks before
the TC test period begins and at the conclusion of TC new release
testing.
Verizon may also develop specific progression test
scenarios for the major changes in functionality of the new release. If
additional progression test scenarios are developed, they will be
distributed through BA Change Control two weeks prior to the start of TC
testing. The progression test scenarios will be posted to the Verizon
Wholesale Web site shortly after their distribution through BA Change
Control.
The progression test scenarios will also be run in
the CLEC Test Environment at the same time as the Quality Baseline
Validation Test Decks are run in the CLEC Test Environment. Results will
be reported with those of the Quality Baseline Validation Test Decks.
After new release testing is concluded, some of the release specific
progression scenarios may be moved into the Quality Baseline Validation
Test Decks.
For Pre-Order transactions, the test decks consist
of inbound requests and corresponding outbound responses. For Order
transactions, the test decks consist of the LSR, the inbound EDI
request, and the outbound EDI response (e.g., confirmation).
These test deck scenarios and test deck accounts
are available for TCs to use during the testing period. However, TCs are
not limited to these transactions and accounts and may request
additional support from Verizon to build specific test accounts in the
CLEC Test Environment. Such requests must be received as part of a test
plan two weeks prior to the beginning of TC testing.
TCs may also request that some of their accounts
that currently reside in production be moved into the CLEC Test
Environment to be available for testing. Generic TC accounts in the CLEC
Test Environment may be duplicated using the identification of another
TC.
Getting
Ready for the New Release Testing
TCs are notified of the content of the release
through the change management process. TCs should review the content of
the release and determine if they want to participate in the test and
what transactions they would like to submit as part of the test.
Verizon will put out an industry notification
requesting TCs to identify their intent to participate in the test based
on the schedule at the end of this section. At that time, Verizon will
publish any changes to the schedule.
TCs wishing to participate in the test should make
arrangements with the CLEC Testing coordinator.
As identified on the schedule, TCs need to submit
a test plan identifying the nature and volumes of transactions they
intend to submit as part of the test and when the transactions will be
submitted. The test coordinator will work with the TC in determining the
appropriate testing scenarios, and to ensure that the test plan will
meet the TC requirements to test the new release, and if additional
accounts must be added to the CLEC Test Environment database based on
their test plan.
New
Release Testing Process
Four weeks prior to a TC impacting release, code
for that release will be loaded into the CLEC Test Environment. This
code will already have gone through Quality Assurance testing by
Verizon. Verizon will run the test decks through the new release system
code and publish the results of this test through BA Change Control. The
published results will indicate if there were any differences from what
was documented in the test decks previously distributed through BA
Change Control.
TCs will begin new release testing on the Monday
four weeks before release implementation and may submit test
transactions normally between 8:00 a.m. and 5:00 p.m. Eastern Monday
through Friday. Verizon may offer extended hours, or changes to the
schedule, and any such changes will be communicated through the Change
Management Process. The CLEC Test Environment will be unavailable for
new release testing the Friday before release implementation into the
CLEC Test Environment and into production. Orders that qualify for level
5 transactions will flowthrough to the service order processor.
Acknowledgements, confirmations, and (where mutually agreed) completions
will be provided. Order transactions that do not flowthrough will be
manually entered into the service order processor and the same responses
provided.
During new release testing, the testing
coordinator will be the TCs point of contact to identify and resolve
problems and questions. The testing coordinator will involve other
Verizon personnel as required. Verizon will maintain a log of issues
during the new release test. Issues will be responded to by the next
business day. The status of open issues will be published and reviewed
with the TCs on status calls to be held by the TC Testing Director, or
designate, every Tuesday and Friday.
On the last Monday of the TC new release testing
period, a special status call will be held to identify any outstanding
issues that must be fixed prior to release implementation.
Verizon will not make any changes to the CLEC Test
Environment while TCs are testing the new release. Defects will be fixed
each Wednesday evening during TC new release testing. Emergency fixes
may be implemented at times other than Wednesday evening. The last
Wednesday will be reserved for fixes that must be done prior to release
implementation Any of these additional fixes will be communicated
through Change Management Process. This enables the TCs an opportunity
to retest before the code is migrated to production.
The escalation procedure to be used, if necessary,
to resolve issues during TC new release testing is at the end of this
section.
Post New
Release Testing
After completion of TC testing, the code will be
migrated into production. The CLEC Test Environment will continue to
contain this code until the code for the next release is moved into the
CLEC Test Environment.
Verizon will execute the test decks in the CLEC
Test Environment at the end of the new release testing period and verify
that the results match the published test decks. After the code
contained in the CLEC Test Environment has been migrated to production,
Verizon will run the Quality Baseline Validation Test Decks in
production without changing the end state of accounts; and Verizon will
document the results within 5 days. Completions will not be a part of
the results obtained from production.
After each release has been moved into production,
Verizon will affirm that it has used software configuration management
tools to ensure that the CLEC Test Environment code was successfully
moved into production.
Sample
New Release Testing Schedule
Monday 8 weeks prior to release implementation
Verizon will request the TC's to
indicate their intent to participate in Release testing via Change
Management Process
Monday 6 weeks prior to release
implementation
TCs will provide an initial Release Test plan to their test coordinator
Monday 6 weeks prior to release
implementation
Verizon publishes test deck scenarios developed for the release. TCs
provide test plan and account requirements to Verizon.
Friday 4 weeks prior to release implementation
CLEC Test Environment unavailable.
Release migrated to CLEC Test Environment. Test
decks run in CLEC Test Environment.
Weekend 4 weeks prior to release implementation
Verizon publishes results of test decks run in the CLEC Test Environment.
Monday 4 weeks prior to release implementation
TC testing begins.
Tuesday/Friday each of the 4 weeks before release
implementation
Status calls are held.
Wednesday each of the 4 weeks before release
implementation
Code fixes are implemented after 5:00PM as needed.
Monday 1 week prior to release implementation
Special status call is held.
Thursday before release implementation
TC testing concludes.
Weekend of release implementation
Test decks run in CLEC Test Environment and results published.
Release migrated to production.
Verizon verifies code successfully migrated.
Monday
Test decks run in production and results published within 5 days.
Escalation Process
- Purpose. To provide processes for TCs to raise
and address issues arising during quality assurance testing of new
releases with Verizon.
- Prerequisites to Escalations. The expectation is
that escalations should occur only after reasonable efforts have been
made to resolve the issue with Verizon's testing team and test
coordinator.
- Escalation by a TC during Testing.
- If reasonable efforts to resolve an issue with
Verizon fail, an individual TC (or group of TCs acting jointly) may
escalate the issue to the TC Testing Director.
- The escalation and the subsequent responses and
replies should take the form of e-mails, with copies to the industry
change control distribution. The TC will also call the TC Testing
Director to provide notification that an e-mail was sent.
- The escalation should include the following:
An explanation of the issue.
A brief history of the steps taken to resolve the
issue with Verizon's testing team and testing manager.
The desired outcome of the escalation.
The impact to the TC if the issue is not resolved.
Contact information, including name, title, phone
number, and e-mail address.
- The Director will acknowledge receipt of an
escalation. The Director will provide the escalating TC with an
initial finding within 1 business day and a final response within 2
business days.
- The initial finding should include the following:
- A brief explanation of the issue, if different from that provided by
the escalating TC.
A brief statement of the likely resolution of the issue.
- The response should include the following:
An explanation of the issue, if different from that provided by the
escalating TC.
A description of the steps taken by Verizon to resolve the
escalation.
A proposed resolution of the issue.
- The escalating TC must reply to the Director's
response within 2 business days, informing Verizon whether the TC
intends to escalate the issue further.
- If the Director fails to resolve the issue to the
TCs satisfaction, the TC may escalate the issue to the Vice-President
- Wholesale Customer Support over the TC Testing Process.
- The second escalation need not repeat
information, but it should describe any developments subsequent to the
first escalation, including a copy of the director's response.
- The Vice President will provide a final response
within 1 business day.
- The reply need not repeat information, but it
should describe any developments subsequent to the first escalation.
- If unsatisfied with an outcome, either party can
seek appropriate relief.
Sample
PON Tracking Spreadsheet
TC Test Case # Version Description PON # Date/Time
Sent To Verizon Pass/ Fail Issues
- AA Change Hunting from C to S, Add additional DL
TESTPON01 3/22/99 6:30
p.m. Fail There should not be an ALI on second DL form because LACT is N
AB
4/01/99 10:00 a.m. Fail
REQTYP missing AC Pass
LSC sent
- AA Change Hunting from P to M, change to Non-Pub
TESTPON02 3/22/99 6:30
p.m. Pass LSC sent
- AA Remove Foreign Listing TESTPON03 3/28/99
1:00 p.m. Fail Failed in EDI for
incorrect EDI sequence. Please review LSR to insure EDI segments in
correct order. AB 3/30 11:50
a.m. Fail Initiator ID and Initiator tel no not sent.
View
entire OSS Interface Change Management Process Document
|