*************************************************** NOTICE *************************************************** This document was converted from Word97 to ASCII Text format. Content from the original version of the document such as headers, footers, footnotes, endnotes, graphics, and page numbers will not show up in this text version. All text attributes such as bold, itallic, underlining, etc. from the original document will not show up in this text version. Features of the orginal document layout such as columns, tables, line and letter spacing, pagination, and margins will not be preserved in the text version. If you need the complete document, download the Word97, or Adobe Acrobat versions, if available. The path and name of the Word97 and Acrobat files will be the same as the ASCII Text file except that they will end with the letters wp, doc, or pdf respectively, instead of the letters txt. ***************************************************** Before the Federal Communications Commission Washington, D.C. 20554 In the Matter of Application by Bell Atlantic New York for Authorization Under Section 271 of the Communications Act To Provide In-Region, InterLATA Service in the State of New York ) ) ) ) ) ) ) ) ) CC Docket No. 99-295 MEMORANDUM OPINION AND ORDER Adopted: December 21, 1999 Released: December 22, 1999 By the Commission: Chairman Kennard and Commissioners Ness and Powell issuing separate statements; Commissioner Furchtgott-Roth concurring and issuing a statement. TABLE OF CONTENTS Paragraph I. INTRODUCTION AND OVERVIEW 1 II. BACKGROUND 17 A. STATUTORY FRAMEWORK 17 B. HISTORY OF THIS APPLICATION 21 C. NEW YORK COMMISSION AND DEPARTMENT OF JUSTICE EVALUATIONS 23 III. ANALYTICAL FRAMEWORK 29 A. ABSENCE OF UNBUNDLING RULES 29 B. SCOPE OF EVIDENCE IN THE RECORD 32 1. Procedural Framework 32 2. Motions To Strike 38 3. Ex Parte Submissions 41 C. FRAMEWORK FOR ANALYZING COMPLIANCE WITH STATUTORY REQUIREMENTS 43 1. Legal Standard 44 2. Evidentiary Case 47 IV. COMPLIANCE WITH SECTION 271 (C)(1)(A) 61 A. BACKGROUND 61 B. DISCUSSION 62 V. COMPLIANCE WITH CHECKLIST 63 A. CHECKLIST ITEM 1 - INTERCONNECTION 63 1. Non-Pricing Aspects of Interconnection 63 2. Pricing of Collocation 77 B. CHECKLIST ITEM 2 - UNBUNDLED NETWORK ELEMENTS 81 1. Operations Support Systems 82 2. Combinations of Unbundled Network Elements 229 3. Pricing of Network Elements 237 C. CHECKLIST ITEM 3 - POLES, DUCTS, CONDUITS, AND RIGHTS-OF-WAY 263 1. Background 263 2. Discussion 265 D. CHECKLIST ITEM 4-UNBUNDLED LOCAL LOOPS 268 1. Background 268 2. Discussion 273 E. CHECKLIST ITEM 5 -- UNBUNDLED LOCAL TRANSPORT 337 1. Background 337 2. Discussion 338 F. CHECKLIST ITEM 6 - UNBUNDLED LOCAL SWITCHING 343 1. Background 343 2. Discussion 346 G. CHECKLIST ITEM 7 349 1. 911 and E911 Access 349 2. Directory Assistance/Operator Services 351 H. CHECKLIST ITEM 8 - WHITE PAGES DIRECTORY LISTINGS 357 1. Background 357 2. Discussion 360 I. CHECKLIST ITEM 9 - NUMBERING ADMINISTRATION 362 1. Background 362 2. Discussion 364 J. CHECKLIST ITEM 10 - DATABASES AND ASSOCIATED SIGNALING 365 1. Background 365 2. Discussion 366 K. CHECKLIST ITEM 11 - NUMBER PORTABILITY 367 1. Background 367 2. Discussion 369 L. CHECKLIST ITEM 12 - LOCAL DIALING PARITY 372 1. Background 372 2. Discussion 374 M. CHECKLIST ITEM 13 -- RECIPROCAL COMPENSATION. 375 1. Background 375 2. Discussion. 376 N. CHECKLIST ITEM 14 - RESALE 378 1. Background 378 2. Discussion 381 3. Provisioning 400 VI. SECTION 272 COMPLIANCE 401 A. BACKGROUND 401 B. DISCUSSION 403 1. Structural, Transactional, and Accounting Requirements of Section 272 404 2. Nondiscrimination Safeguards of Section 272 417 3. Joint Marketing Requirements of Section 272 419 VII. PUBLIC INTEREST ANALYSIS 422 A. OVERVIEW 422 B. COMPETITION IN LOCAL EXCHANGE AND LONG DISTANCE MARKETS 425 1. Impact on Local Competition 426 2. Impact on Long Distance Competition 428 C. ASSURANCE OF FUTURE COMPLIANCE 429 1. Summary of Performance Reporting and Enforcement Mechanisms 431 2. Key Elements of the Enforcement Plan 433 D. OTHER ARGUMENTS 444 VIII. SECTION 271(D)(6) ENFORCEMENT AUTHORITY 446 IX. CONCLUSION 454 X. ORDERING CLAUSES 455 APPENDIX A: LIST OF COMMENTERS APPENDIX B: STATISTICAL METHODOLOGY APPENDIX C: ANALYSIS OF AVERAGE COMPLETED INTERVALS FOR NON- DISPATCH ORDERS USING CARRIER TO CARRIER AND GERTNER/BAMBERGER STUDY DATA I. INTRODUCTION AND OVERVIEW 1. In this Order, we grant Bell Atlantic's application to enter the interLATA long distance market in New York State based on our conclusion that Bell Atlantic has taken the statutorily required steps to open its local exchange and exchange access markets to competition. The market opening actions by the New York Commission and Bell Atlantic underlying our decision bring the telecommunications industry one step closer to realization of the full pro- competitive goals of the 1996 Telecommunications Act, and promise substantial benefits for consumers in the form of lower rates and innovative service packages. Bell Atlantic filed the application addressed in this Order with the Commission on September 29, 1999. Fifty-seven parties filed comments on the application on October 18, 1999. Of these, more than twenty parties supported grant of the application. Twenty-five parties filed reply comments on November 8, 1999. 2. Our decision today approving Bell Atlantic's application represents the culmination of extensive federal and state efforts implementing the Telecommunications Act of 1996. This action builds on the experience that this Commission has gained from reviewing prior section 271 applications and developing rules to implement section 251 of the Communications Act. Significantly, it also builds on the tireless efforts of the New York Commission, which has worked long and hard with Bell Atlantic and competitive local exchange companies (LECs) to ensure that local markets in New York are open to competition. 3. In enacting the telephony provisions of the 1996 Act, Congress envisioned fundamental pro-competitive changes in the then-existing telecommunications environment. To this end, Congress took the momentous step of requiring that the incumbent LECs open the traditionally non-competitive local exchange and exchange access markets to competition in order to foster the entry of alternative service providers. Once the Bell Operating Companies (BOCs) have opened their local markets to competition, the 1996 Act permits them to enter the in-region, interLATA toll market, thereby increasing competition in the long distance telecommunications market. 4. Unfortunately, implementation of this congressional vision of increased telecommunications competition has, in many instances, not proceeded swiftly or smoothly. For example, some of the section 271 applications that we have reviewed to date have fallen far short of the statutory requirements. Moreover, some carriers attacked sections 271-275 of the Act on constitutional grounds arguing that each constitutes an impermissible bill of attainder. The court roundly rejected this challenge, stating that these provisions "are constitutionally sound." We believe that the instant application represents a turning point in the process of implementing the 1996 Act, with a new focus by the BOCs on taking the steps necessary to open the local exchange and exchange access markets to competition. 5. While this is the first section 271 application to receive Commission approval, our decision here reflects the fundamental principles adopted in our prior section 271 orders. Thus, we apply the general standards developed in prior orders in evaluating section 271 compliance - whether the BOC is providing service to competitors at parity with its retail offerings or, when there is no analogous retail activity, whether the BOC's performance would allow an efficient competitor a meaningful opportunity to compete. Based on our growing experience in addressing issues involving the development of local exchange competition, we also apply these standards in a pragmatic fashion, thus building on our prior decisions. For example, we consider the overall picture presented by the record, rather than focusing on any one aspect of performance. 6. It is no coincidence that this historic first is recorded in New York, a state that has been a leader in opening local markets to competition for over fifteen years, and a state with one of the most rigorous, expert commissions in the nation. Without the dedicated work and unfailing persistence of the New York Commission over the past several years, it is unlikely that this application would have reached a point at which it merits approval. It is also noteworthy that New York State has some of the most intensely competitive local exchange and exchange access markets in the nation. This track record of successful competition places the present application in a different context from prior filings. For the first time, we can evaluate compliance with the requirements of section 271 in a market context, rather than relying solely on predictive judgment. 7. We applaud the dedicated efforts of the New York Commission, beginning shortly after passage of the 1996 Act, to work with Bell Atlantic and competitive LECs to ensure that Bell Atlantic would achieve compliance with section 271. A number of the parties to this proceeding also praise the work of the New York Commission. Even AT&T, which strongly opposes the application, agrees that the New York Commission has significantly advanced Bell Atlantic's progress toward compliance with section 271. MCI states that "[a]t the insistence of the New York State Public Service Commission . . . BA-NY has done much to open its local markets . . . " Nextlink, one of the competitors supporting the application, also cites with approval the "open, collaborative process that included independent third party testing, numerous industry workshops, and staff solicitation and review of detailed public comments." 8. The section 271 process in New York exemplifies the way in which rigorous state proceedings can contribute to the success of a section 271 application. There are a number of elements that were particularly important to the success of this process in opening local markets to competition consistent with the terms of the 1996 Act. These include: (1) full and open participation by all interested parties; (2) extensive independent third party testing of Bell Atlantic's operations support systems (OSS) offering; (3) development of clearly defined performance measures and standards; and (4) adoption of performance assurance measures that create a strong financial incentive for post-entry compliance with the section 271checklist by Bell Atlantic. While we accord applicants flexibility in demonstrating compliance with section 271, these elements played a vital role in the success of this application. 9. First, under the auspices of the New York Commission, both competitive LECs and Bell Atlantic participated fully in collaborative sessions and technical workshops to clarify or resolve issues. This ensured broad-based industry participation throughout the proceeding. 10. Second, extensive third party testing of Bell Atlantic's OSS in New York was also critical to the success of these proceedings. The OSS testing was conducted in two phases. Phase I consisted of development of a detailed and comprehensive plan to evaluate and test the OSS interfaces and the adequacy of Bell Atlantic's processes, procedures, and documentation to allow competitive LECs to access and use these systems. Phase II of the test involved: (1) building the interface and assessing the ease or complexity of developing interface software; and (2) executing the test plan using a pseudo-competitive LEC. The rigorous, comprehensive third party testing in New York identified numerous shortcomings in Bell Atlantic's OSS performance that were subsequently corrected and re-tested. KPMG released its final report on August 6, 1999, concluding that Bell Atlantic's OSS was commercially available and sufficient to handle reasonable, anticipated commercial volumes. 11. Third, the New York Commission developed, and continues to refine, inter-carrier performance measures and service quality standards in its Carrier-to-Carrier proceeding. For example, the New York Commission has instituted collaborative proceedings to address xDSL issues and is developing xDSL specific performance measures and standards. This effort represents an ongoing process as a number of additional standards remain under development. To ensure that the company's performance data or "metrics" are reported reliably in accordance with the New York Commission's definitions, New York staff and KPMG reviewed the adequacy of internal controls surrounding the data collection process. In addition, the New York Commission's staff verifies on a monthly basis that Bell Atlantic's reported results conform to the definitions developed in the Carrier-to-Carrier proceeding. The definitions and standards developed in that proceeding have done much to foster the development of consistent and meaningful data concerning Bell Atlantic's performance. This gives us greater confidence that our decision is based on performance data that accurately measures Bell Atlantic's actual performance. 12. Fourth, the New York Commission has adopted Bell Atlantic's proposal for self- effectuating performance assurance plans that will provide significant financial incentives for Bell Atlantic to maintain an open market and prevent "backsliding" in the future provision of service by Bell Atlantic to competitive LECs. It is important that these plans are designed to function automatically without imposing administrative and regulatory burdens on competitors. It is also significant that the New York Commission is committed to supervising the implementation of these plans. 13. The well established pro-competitive regulatory environment in New York in conjunction with recent measures to achieve section 271 compliance has, in general, created a thriving market for the provision of local exchange and exchange access service. Competitors in New York are able to enter the local market using all three entry paths provided under the Act. These new entrants are serving both residential and business customers in geographic areas throughout the state, although competition is most intense for business customers in urban areas, especially in New York City. As a result, the extent of competition in New York greatly exceeds that in the other states for which BOCs have filed section 271 applications. 14. Bell Atlantic estimates that competitors serve at least 1,118,180 lines in New York. According to Bell Atlantic, competitors serve at least 651,793 lines using their own facilities, 152,055 lines using the UNE platform, and 314,332 lines through resale. Bell Atlantic states that competitive LECs serve both residential and business customers. Bell Atlantic estimates that competitors in New York serve at least 35,753 residential lines over their own facilities. In addition, Bell Atlantic estimates that competitive LECs in New York provide service to 137,342 residential customers using the UNE platform and resell another 63,547 residential lines. Similarly, Bell Atlantic estimates that competitive LECs in New York serve at least 612,000 business customers over their own facilities. Competitive LECs serve an additional 14,713 business lines using the UNE platform and resell another 250,785 business lines. 15. Our action today clearly demonstrates that when a BOC takes the steps required to open its local markets to full competition, the company will be rewarded with section 271 authority to enter the long distance market. The market opening requirements of the 1996 Act demand substantial changes in the way the BOCs have historically done business, and opening the New York market to full local competition has not been an easy process for Bell Atlantic or the New York Commission. We commend their hard work in reaching this historic achievement. 16. Finally, we wish to emphasize that grant of this application may close this chapter of the proceeding, but it is not the end of the story. Bell Atlantic must continue to comply with the checklist requirements, and with the requirements of section 272 of the Act. Section 271(d)(6) provides specific tools that augment our preexisting enforcement authority, to be used if Bell Atlantic falls out of compliance with the conditions required for grant of its application. Most notably, section 271(d)(6) authorizes the Commission to suspend or revoke the authorization granted here. This is a powerful enforcement tool, which should create a strong incentive for Bell Atlantic to ensure that its performance does not diminish. We expect that Bell Atlantic will not risk facing the severe remedy of having its authority to market service suspended, but stress that we are prepared to use this remedy if Bell Atlantic's performance in implementing the checklist deteriorates. II. BACKGROUND A. Statutory Framework 17. In the 1996 Act, Congress conditioned BOC provision of in-region, interLATA service on compliance with certain provisions of section 271. Pursuant to section 271, BOCs must apply to this Commission for authorization to provide interLATA services originating in any in-region state. Congress has directed the Commission to issue a written determination on each application no later than 90 days after the application is filed. 18. To obtain authorization to provide in-region, interLATA services under section 271, the BOC must show that: (1) it satisfies the requirements of either section 271(c)(1)(A), known as "Track A" or 271(c)(1)(B), known as "Track B"; (2) it has "fully implemented the competitive checklist" or that the statements approved by the state under section 252 satisfy the competitive checklist contained in section 271(c)(2)(B); (3) the requested authorization will be carried out in accordance with the requirements of section 272; and (4) the BOC's entry into in- region, interLATA market is "consistent with the public interest, convenience, and necessity." The statute specifies that unless the Commission finds that these four criteria have been satisfied, the Commission "shall not approve" the requested authorization. 19. Section 271(d)(2)(A) requires the Commission to consult with the Attorney General before making any determination approving or denying a section 271 application. The Attorney General is entitled to evaluate the application "using any standard the Attorney General considers appropriate," and the Commission is required to "give substantial weight to the Attorney General's evaluation." Section 271(d)(2)(A) specifically provides, however, that "such evaluation shall not have any preclusive effect on any Commission decision." Thus, Congress clearly contemplated that, in some circumstances, the Commission could reach a different conclusion from the Department, even after giving "substantial weight" to the Department's views. 20. In addition, the Commission must consult with the relevant state commission to verify that the BOC has one or more state approved interconnection agreements with a facilities- based competitor, or a statement of generally available terms and conditions (SGAT), and that either the agreement(s) or general statement satisfy the "competitive checklist." In the Ameritech Michigan Order, the Commission determined that, because the Act does not prescribe any standard for Commission consideration of a state commission's verification under section 271(d)(2)(B), it has discretion in each section 271 proceeding to determine the amount of weight to accord to the state commission's verification. The Commission has held that, although it will consider carefully state determinations of fact that are supported by a detailed and extensive record, it is the Commission's role to determine whether the factual record supports the conclusion that particular requirements of section 271 have been met. In the instant proceeding, we accord the New York Commission's evaluation substantial weight, for the reasons set forth above. In particular, we note that the New York Commission has directed a rigorous collaborative process that has included: an extensive independent third-party test of Bell Atlantic's OSS interfaces, processes and procedures; active participation by New York Commission staff, Bell Atlantic, and competitive LECs in numerous technical conferences that helped to identify and resolve problems; and the development of a comprehensive performance monitoring and enforcement mechanism. Throughout these proceedings, the New York Commission has ensured that the process was open to participation by all interested parties and, as a result, received and reviewed a massive record of public comments. We thus place substantial weight on the New York Commission's conclusions, as they reflect its role not only as a driving force behind these proceedings, but also as an active participant in bringing local competition to the state's markets. B. History of this Application 21. On February 13, 1997, Bell Atlantic, filed a draft application under section 271, along with a Statement of Generally Applicable Terms and Conditions with the New York Commission. On July 8, 1997, after a number of technical conferences and collaborative meetings and technical and legal analyses, a New York Commission Administrative Law Judge concluded that Bell Atlantic had made a prima facie case regarding certain offerings, but had not met its burden of proof regarding commercial availability, procedure standardization, timeliness, and measuring parity. Subsequently, the New York Commission held additional collaborative sessions to work out technical details associated with development of a working Operations Support System (OSS). Specifically, these sessions resolved numerous OSS issues, including an agreement on business rules that would govern the development by competitors of systems to interface with those of Bell Atlantic. Following approval of the Bell Atlantic/NYNEX merger, Bell Atlantic filed a supplemental section 271 application with the New York Commission, which was followed by additional filings and technical conferences. After completion of this process, Bell Atlantic agreed to make additional commitments in connection with its application for section 271 approval. 22. On April 6, 1998, Bell Atlantic filed a Pre-Filing Statement with the New York Commission, which contained a number of commitments, including: 1) to provide combinations of elements (including UNE-P as a minimum service offering); 2) to engage a third-party to test Bell Atlantic's OSS; and 3) to establish a self-effectuating system to prevent backsliding. Pursuant to these commitments, Bell Atlantic obtained a comprehensive independent third-party test of its wholesale support systems and developed a plan to ensure adequate continuing wholesale performance. As described above, this test was conducted by KPMG Peat Marwick and Hewlett Packard under the supervision of the New York Commission. Together, the New York Commission and KPMG created an open testing environment in which they consulted with interested parties, issued draft plans and reports, and reported in detail on issues of serious concern. The problems identified through the test were addressed by Bell Atlantic through process improvements during the test period. The third-party test was completed with the release of KPMG's final report on August 6, 1999. As noted above, Bell Atlantic filed its application with this Commission on September 29, 1999. C. New York Commission and Department of Justice Evaluations 23. On October 18, 1999, the New York Commission submitted to this Commission its evaluation of Bell Atlantic's application. The New York Commission advised the Commission that, following two and half years of review, testing, and process improvements, Bell Atlantic-NY had met the checklist requirements of section 271(c). Specifically, New York stated that Bell Atlantic had met its obligation under section 271(c)(1)(A) by entering into more than 75 interconnection agreements approved by the New York Commission, and that competitive LECs are providing local exchange service in New York using their own facilities and those of Bell Atlantic. In addition, the New York Commission stated that the record developed in the New York proceeding establishes that Bell Atlantic has a legal obligation, under its interconnection agreements and state-approved tariffs, to provide the 14 items required under section 271's checklist, and that Bell Atlantic is meeting its legal obligation to provide those 14 items. 24. On November 1, 1999, the Department of Justice filed its evaluation. Consistent with its approach in past applications, the Department stated that it considers whether all three entry paths contemplated by the 1996 Act - facilities-based entry involving construction of new networks, the use of unbundled elements of the BOC's network, and resale of the BOC's services - are fully and irreversibly open to competitive entry to serve both business and residential customers. The Department of Justice found that "Bell Atlantic has completed most - but not all - of the actions needed to achieve a fully and irreversibly open market in New York." The Department concluded that it did not have substantial concerns about the ability of facilities-based carriers and firms that wish to resell Bell Atlantic's retail services to enter the local telecommunications markets in New York. It also concluded that Bell Atlantic has made "great progress in opening the market to competition through the use of unbundled network elements," but two major areas of deficiency-OSS and access to local loops - remain as important obstacles to local competition. The Department of Justice also concluded, however, that Bell Atlantic has not yet demonstrated that it can adequately provide access to unbundled local loops, either for traditional voice services or for digital subscriber line (DSL) technology used to provide a variety of advanced services. Moreover, the Department expressed concern that Bell Atlantic's systems for handling orders for the unbundled network platform rely on manual processes that are prone to error and delay. The Department expressly reserved judgment, however, on whether the facts in the record established compliance with the legal requirements of the competitive checklist or the Commission's rules. 25. The Department of Justice stated its belief that its assessment of the facts regarding Bell Atlantic's wholesale performance was substantially consistent with the New York's assessment. The Department of Justice noted that, to the extent there is a difference between its evaluation and that of the New York Commission, "it arises largely from the Department's conclusion that needed improvements should be achieved before Bell Atlantic is authorized to provide interLATA services in New York, rather than relying on post-271 approval mechanisms to attempt to ensure such improvements." 26. The Department urged us not to permit Bell Atlantic to offer interLATA services until "it demonstrates that it has solved the existing problems in its provision of access to unbundled network elements." It noted, however, that it "is possible that information from Reply Comments and ex parte submissions will provide additional support for Bell Atlantic's claims and justify a conclusion different from that reached by the Department on the basis of the current record." 27. The Department of Justice stated that this Commission could properly deny this application. As an alternative, the Department suggested the Commission might be able to approve the application subject to carefully crafted conditions "under which Bell Atlantic would be permitted to offer interLATA services only after taking specified steps and demonstrating that its performance has met appropriate requirements." The Department of Justice thus concluded that "the Commission may be able to approve Bell Atlantic's application at the culmination of these proceedings." 28. On November 8, 1999, the New York Commission, and 23 other parties, filed reply comments in this proceeding. Both Bell Atlantic and the New York Commission contended that the arguments raised in opposition are insufficient grounds for denying the application. III. ANALYTICAL FRAMEWORK A. Absence of Unbundling Rules 29. It is necessary to clarify, for the purpose of evaluating this application, which network elements we expect Bell Atlantic to demonstrate that it provides on an unbundled basis, pursuant to section 251(c)(3) and checklist item 2. In the Local Competition First Report and Order, the Commission established a list of seven UNEs which incumbent LECs were obliged to provide: (1) local loops; (2) network interface devices; (3) local and tandem switching; (4) interoffice transmission facilities; (5) signaling networks and call-related databases; (6) operations support systems; and (7) operator services and directory assistance. This obligation was codified in section 51.319 of the Commission's rules ("rule 319"). In January 1999, the Supreme Court vacated rule 319 and instructed the Commission to revise the standards under which the unbundling obligation is determined and to reevaluate the network elements subject to the unbundling requirement. 30. Although the former rule 319 was not in force at the time Bell Atlantic filed its application in this proceeding, Bell Atlantic has sought to demonstrate that it provides nondiscriminatory access to these network elements. Indeed, Bell Atlantic has stated that it believes it would be "reasonable" for the Commission to use the original seven network elements identified in former rule 319 in evaluating this application. In assessing Bell Atlantic's argument, we begin from the premise that compliance with the competitive checklist requires that Bell Atlantic provide nondiscriminatory access to network elements, as contemplated by, and in accordance with, the requirements of sections 251(c)(3) and 251(d)(2). We believe that using the network elements identified in former rule 319 as a standard in evaluating Bell Atlantic's application, during the interim period between its vacation by the Supreme Court and the effective date of the new rules, is a reasonable way to ensure that the application complies with the checklist requirements. We find it significant that no commenter has taken the position in this proceeding that Bell Atlantic should not be required to demonstrate that it provides these network elements. Accordingly, for the purposes of this application, we will evaluate whether Bell Atlantic provides nondiscriminatory access to the seven network elements identified under former rule 319. 31. We disagree with commenters that contend that Bell Atlantic must demonstrate, for the purposes of this application, compliance with the rules governing unbundled network elements recently established in the UNE Remand proceeding. These new rules, among other things, specify which network elements an incumbent LEC is obliged to unbundle, and establish several new obligations that were not present under the former rule 319. We recognize, however, that these new rules will not take effect until some time after release of this order. Therefore, we will not require Bell Atlantic to prove that it currently complies with rules that have yet to take effect. Moreover, we believe it would be inequitable to require Bell Atlantic to comply with these rules, particularly when no other incumbent LEC must comply before the effective date, just because Bell Atlantic has a section 271 application pending before the Commission. Of course, the Commission expects that Bell Atlantic will comply with the new UNE Remand rules once they take effect. B. Scope of Evidence in the Record 1. Procedural Framework 32. Section 271 proceedings are, at their core, adjudications that the Act requires the Commission to complete within ninety days of the application filing. The statute also requires us to consult with the Department of Justice and the relevant state commission in reviewing the application. 33. In the context of this statutory framework, the Commission has established procedural rules governing BOC section 271 applications. Among other things, these rules provide an opportunity for parties other than the Department of Justice and the relevant state commission to comment on section 271 applications. 34. Under our procedural rules governing BOC section 271 applications, we expect that a section 271 application, as originally filed, will include all of the factual evidence on which the applicant would have the Commission rely in making its findings. An applicant may not, at any time during the pendency of its application, supplement its application by submitting new factual evidence that is not directly responsive to arguments raised by parties commenting on its application. This includes the submission, on reply, of factual evidence gathered after the initial filing. In an effort to meet its burden of proof, however, a BOC may submit new factual information after the application is filed, if the sole purpose of that evidence is to rebut arguments or facts submitted by other commenters. The new evidence, however, must cover only the period placed in dispute by commenters and may, in no event, post-date the filing of the comments (i.e., day 20). In the event that the applicant submits new or post-dated evidence in replies or ex parte filings, we retain the discretion to start the 90-day review process anew or to accord such evidence no weight. 35. This precedent has served the Commission well, by deterring incomplete filings from the BOCs. In particular, the rule is designed to prevent applicants from presenting part of their initial prima facie showing for the first time in reply comments. The rule has enabled us properly to manage our own internal consideration of the application and ensures that commenters are not faced with a "moving target" in the BOC's section 271 application. We continue to believe, as a general matter, that it is highly disruptive to our processes to have a record that is constantly evolving. We emphasize, however, that our precedent makes clear that this rule is a discretionary one. 36. We do not expect that a BOC, in its initial application, will anticipate and address every foreseeable argument its opponents might make in their subsequent reply comments, but we have previously stated that a BOC must address in its initial application all facts that the BOC can reasonably anticipate will be at issue. Through state proceedings, BOCs should be able reasonably to identify and anticipate certain arguments and allegations that parties will make in their filings before the Commission. 37. In addition, the Commission has found that a BOC's promises of future performance to address particular concerns raised by commenters have no probative value in demonstrating its present compliance with the requirements of section 271. In order to gain in- region, interLATA entry, a BOC must support its application with actual evidence demonstrating its present compliance with the statutory conditions for entry, instead of prospective evidence that is contingent on future behavior. Thus, we must be able to make a determination based on the evidence in the record that a BOC has actually demonstrated compliance with the requirements of section 271. 2. Motions To Strike 38. On November 22, 1999, AT&T filed a motion to strike or to disregard portions of the reply submissions of Bell Atlantic and the New York Commission filed in this proceeding. AT&T argues that reply submissions of both Bell Atlantic and the New York Commission contain material that must be stricken or accorded no weight under the Commission's rules because they post-date Bell Atlantic's application and the due date for comments. In addition, AT&T argues that Bell Atlantic's reply submission contains numerous new promises of future performance. 39. We deny AT&T's motion because we do not rely, as a basis for our decision, on: (1) evidence submitted by Bell Atlantic after filing its application, unless such evidence both relates to events that occurred prior to the comment filing date (October 19, 1999) and is directly responsive to allegations in the record; (2) evidence submitted by the New York Commission that post-dates the comment due date; or (3) Bell Atlantic's promises of future compliance. 40. On December 17, 1999, Covad filed a motion to strike an ex parte submission filed by Bell Atlantic on December 10, 1999. We deny Covad's motion because we do not rely on Bell Atlantic's ex parte submission as a basis for our decision. 3. Ex Parte Submissions 41. Under the procedural rules governing section 271 applications, we strongly encourage parties to set forth their views comprehensively in their formal submissions (i.e., Brief in Support, oppositions, supporting comments, etc.), and not to rely on subsequent ex parte presentations. At the same time, the Commission expressly provided that parties may file ex partes. Our procedural Public Notice thus clearly contemplates that parties may file written ex partes, when appropriate, to clarify the record. We take this opportunity to clarify that like reply comments, ex partes must be directly responsive to arguments raised by parties commenting on the application. Such ex partes may, however, elaborate on, or provide additional explanation or detail in response to requests from Commission staff or in direct response to post-reply ex parte filings. 42. Nothing in our procedural rules or past precedent precludes the Commission and the staff from requesting clarification or an explanation about information or data contained in the filings specified above. Indeed, our procedural Public Notice expressly recognizes that the Commission may request additional information from the applicant, as the page limit for ex partes does not apply to written material filed in response to direct requests from Commission staff. It is critical to the agency's deliberative process that the Commission and staff fully understand the evidence and arguments presented in the BOC's section 271 application, arguments raised in opposition, and responses made by parties on reply. Accordingly, the Commission retains the discretion to request additional information from the applicant or other parties that elaborates on positions set forth in the original application, comments, or reply comments. We emphasize that we are not departing from our view that the applicant should set forth its position in a clear and concise manner in its formal filings. However, it is imperative that, as part of the Commission's deliberative process, we have the ability to engage in an ongoing dialogue with parties to ensure that we have a clear and accurate understanding of the information contained in all formal submissions. C. Framework for Analyzing Compliance with Statutory Requirements 43. In this section, we discuss two aspects of the framework for analyzing compliance with the statutory requirements of section 271. First, we discuss the legal standards we have enunciated in past orders for determining whether a BOC is meeting the statutory nondiscrimination requirements. Second, we discuss the evidentiary requirements of a BOC's section 271 application and, in particular, the types of showings we will find probative in deciding whether a BOC has met the statutory standards. 1. Legal Standard 44. In order to comply with the requirements of section 271's competitive checklist, a BOC must demonstrate that it has "fully implemented the competitive checklist in subsection (c)(2)(B)." In particular, the BOC must demonstrate that it is offering interconnection and access to network elements on a nondiscriminatory basis. Previous Commission orders addressing section 271 applications have elaborated on this statutory standard. First, for those functions the BOC provides to competing carriers that are analogous to the functions a BOC provides to itself in connection with its own retail service offerings, the BOC must provide access to competing carriers in "substantially the same time and manner" as it provides to itself. Thus, where a retail analogue exists, a BOC must provide access that is equal to (i.e., substantially the same as) the level of access that the BOC provides itself, its customers, or its affiliates, in terms of quality, accuracy, and timeliness. For those functions that have no retail analogue, the BOC must demonstrate that the access it provides to competing carriers would offer an efficient carrier a "meaningful opportunity to compete." As we stated in the Ameritech Michigan Order, there may be situations in which a BOC contends that, although equivalent access has not been achieved for an analogous function, the access that it provides is still nondiscriminatory within the meaning of the statute. 45. We do not view the "meaningful opportunity to compete" standard to be a weaker test than the "substantially the same time and manner" standard. Where the BOC provides functions to its competitors that it also provides for itself in connection with its retail service, its actual performance can be measured to determine whether it is providing access to its competitors in "substantially the same time and manner" as it does to itself. Where the BOC, however, does not provide a retail service that is similar to its wholesale service, its actual performance with respect to competitors cannot be measured against how it performs for itself because the BOC does not perform analogous activities for itself. In those situations, our examination of whether the quality of access provided to competitors offers competitors "a meaningful opportunity to compete" is intended to be a proxy for whether access is being provided in substantially the same time and manner and, thus, nondiscriminatory. 46. Finally, we note that a determination of whether the statutory standard is met is ultimately a judgment we must make based on our expertise in promoting competition in local markets and in telecommunications regulation generally. We have not established, nor do we believe it appropriate to establish, specific objective criteria for what constitutes "substantially the same time and manner" or a "meaningful opportunity to compete." We look at each application on a case-by-case basis and consider the totality of the circumstances, including the origin and quality of the information before us, to determine whether the nondiscrimination requirements of the Act are met. Whether this legal standard is met can only be decided based on an analysis of specific facts and circumstances. 2. Evidentiary Case 47. We previously have set forth the analytical framework that we use in assessing whether a BOC has demonstrated compliance with the statutory requirements of section 271. At the outset, we reemphasize that the BOC applicant retains at all times the ultimate burden of proof that its application satisfies all of the requirements of section 271, even if no party files comments challenging its compliance with a particular requirement. 48. The evidentiary standards governing our review of section 271 applications are intended to balance our need for reliable evidence against our recognition that, in such a complex endeavor as a section 271 proceeding, no finder of fact can expect proof to an absolute certainty. While we expect the BOC to demonstrate as thoroughly as possible that it satisfies each checklist item, the public interest standard, and the other statutory requirements, we reiterate that the BOC needs only to prove each element by "a preponderance of the evidence," which generally means "the greater weight of evidence, evidence which is more convincing that the evidence which is offered in opposition to it." 49. As we held in the Second BellSouth Louisiana Order, we first determine whether the BOC has made a prima facie case that it meets the requirements of a particular checklist item. The BOC must plead, with appropriate supporting evidence, facts which, if true, are sufficient to establish that the requirements of section 271 have been met. Once the BOC has made such a showing, opponents must produce evidence and arguments to show that the application does not satisfy the requirements of section 271, or risk a ruling in the BOC's favor. 50. When considering commenters' filings in opposition to the BOC's application, we look for evidence that the BOC's policies, procedures, or capabilities preclude it from satisfying the requirements of the checklist item. Mere unsupported evidence in opposition will not suffice. Although anecdotal evidence may be indicative of systemic failures, isolated incidents may not be sufficient for a commenter to overcome the BOC's prima facie case. Moreover, a BOC may overcome such anecdotal evidence by, for example, providing objective performance data that demonstrate that it satisfies the statutory nondiscrimination requirement. 51. We will look to the state to resolve factual disputes wherever possible. Indeed, we view the state's and the Department of Justice's role to be one similar to that of an "expert witness." Given the 90-day statutory deadline to reach a decision on a section 271 application, the Commission does not have the time or the resources to resolve the enormous number of factual disputes that inevitably arise from the technical details and data involved in such a complex endeavor. Accordingly, as discussed above, where the state has conducted an exhaustive and rigorous investigation into the BOC's compliance with the checklist, we may give evidence submitted by the state substantial weight in making our decision. Although we are statutorily required to accord substantial weight to the Department of Justice's evaluation, in appropriate circumstances, we may conclude that the evidence submitted by a state commission is more persuasive than that submitted by the Department of Justice, particularly if the state has conducted a rigorous analysis of the evidence. 52. To make a prima facie case that the BOC is meeting the requirements of a particular checklist item under section 271(c)(1)(A), the BOC must demonstrate that it is providing access or interconnection pursuant to the terms of that checklist item. In particular, a BOC must demonstrate that it has a concrete and specific legal obligation to furnish the item upon request pursuant to state-approved interconnection agreements that set forth prices and other terms and conditions for each checklist item, and that it is currently furnishing, or is ready to furnish, the checklist item in quantities that competitors may reasonably demand and at an acceptable level of quality. 53. The particular showing required to demonstrate compliance will vary depending on the individual checklist item and the circumstances of the application. We have given BOCs substantial leeway with respect to the evidence they present to satisfy the checklist. Although our orders have provided guidance on which types of evidence we find more persuasive, "we reiterate that we remain open to approving an application based on other types of evidence if a BOC can persuade us that such evidence demonstrates nondiscriminatory treatment and other aspects of the statutory requirements." In past orders we have encouraged BOCs to provide performance data in their section 271 applications to demonstrate that they are providing nondiscriminatory access to unbundled network elements to requesting carriers. We have concluded that the most probative evidence that a BOC is providing nondiscriminatory access is evidence of actual commercial usage. Performance measurements are an especially effective means of providing us with evidence of the quality and timeliness of the access provided by a BOC to requesting carriers. 54. A number of state commissions, including New York, have established a collaborative process through which they have developed, in conjunction with the incumbent and competing carriers, a set of measures, or metrics, for reporting of performance in various areas. Through such collaborative processes, New York has also adopted performance standards for certain functions, typically where there can be no comparable measure based on the incumbent LEC's retail performance. We strongly encourage this type of process, because it allows the technical details that determine how the metrics are defined and measured to be worked out with the participation of all concerned parties. We also strongly support the efforts of state commissions to build and oversee a process that ensures the development of local competition that Congress intended. An extensive and rigorous evaluation of the BOC's performance by the states provides greater certainty that barriers to competition have been eliminated and the local markets in a state are open to competition. 55. We caution, however, that adoption by a state of a particular performance standard pursuant to its state regulatory authority is not determinative of what is necessary to establish checklist compliance under section 271. We recognize that metric definitions and incumbent LEC operating systems will likely vary among states, and that individual states may set standards at a particular level that would not apply in other states and that may constitute more or less than the checklist requires. Therefore, it is unlikely that we will see uniform standards that measure precisely the same BOC conduct across states. At the same time, for functions for which there are no retail analogues, and for which performance benchmarks have been developed with the ongoing participation of affected competitors and the BOC, those standards may well reflect what competitors in the marketplace feel they need in order to have a meaningful opportunity to compete. 56. We emphasize that, because the Commission is statutorily required to determine checklist compliance, we must independently evaluate whether a BOC is fulfilling the nondiscrimination requirements of section 271. Nevertheless, in making our evaluation we will examine whether the state commission has adopted a retail analogue or a benchmark to measure BOC performance and then review the particular level of performance the state has required. If the state commission has made these determinations in the type of rigorous collaborative proceeding described above, we are much more likely to find that they are reasonable and appropriate measures of parity. Accordingly, we are inclined to rely on such standards and measurements in our own analysis but may reach a different conclusion where justified. 57. In the instant proceeding, for example, the New York Commission has determined, through a collaborative process with input from Bell Atlantic and competing carriers, that there are retail analogues for certain functions and performance benchmarks for others. We find this to be a reasonable basis for us to begin our analysis. Under the framework adopted by the New York Commission, Bell Atlantic determines whether any difference in its performance compared to its retail operations is statistically significant, and provides a figure indicating the degree of statistical significance. For measures where the New York Commission has set a performance benchmark, the New York Commission has required Bell Atlantic to provide the metrics for its performance to competing carriers, which can then be compared to the benchmark. 58. In this case, we conclude that to the extent there is no statistically significant difference between Bell Atlantic's provision of service to competitive LECs and its own retail customers, we need not look any further. Similarly, if there is no difference between the Bell Atlantic provision of service to competitive LECs and the performance benchmark, our analysis is done. 59. To the extent there is any statistically significant difference between Bell Atlantic's provision of service to competitive LECs and retail customers or an apparent difference between its provision of service to competitive carriers and the performance benchmarks set by the New York Commission, we will examine the evidence further to make a determination whether the statutory nondiscrimination requirements are met. Thus, we will examine the explanation that Bell Atlantic and other commenters provide about whether these differences provide an accurate depiction of the quality of Bell Atlantic's performance. For instance, we may examine the data on a more disaggregated level, in order to evaluate arguments made by Bell Atlantic that competitive LEC error, or differences in the composition of competitive LEC orders, or sudden changes in the quantity or timing of orders made by competitive LECs, are responsible for the apparent poor performance. We also may examine how many months a variation in performance has existed and what the trend has been in recent months. A steady improvement in performance over time may provide us with an indication that problems are being resolved. It may also provide us with evidence as to whether Bell Atlantic's systems are scaleable and can handle large volumes of orders for services. Finally, in some instances, we may find that statistically significant differences in measured performance may exist, but that such differences have little or no competitive significance in the marketplace. As such, we may deem such differences non-cognizable under the statutory standard. 60. The determination of whether a BOC's performance meets the statutory requirements necessarily is a contextual decision based on the totality of the circumstances and information before us. There may be multiple performance measures associated with a particular checklist item, and an apparent disparity in performance for one measure, by itself, may not provide a basis for finding noncompliance with the checklist. Other measures may tell a different story, and provide us with a more complete picture of the quality of service being provided. Thus, whether we are applying the "substantially same time and manner" standard or the "meaningful opportunity to compete" standard, we will examine whether the differences in the measured performance are large enough to be deemed discriminatory under the statute. IV. COMPLIANCE WITH SECTION 271 (C)(1)(A) A. Background 61. In order for the Commission to approve a BOC's application to provide in-region, interLATA services, a BOC must first demonstrate that it satisfies the requirements of either section 271(c)(1)(A) (Track A) or 271(c)(1)(B) (Track B). To qualify for Track A, a BOC must have interconnection agreements with one or more competing providers of "telephone exchange service . . . to residential and business subscribers." The Act states that "such telephone service may be offered . . . either exclusively over [the competitor's] own telephone exchange service facilities or predominantly over [the competitor's] own telephone exchange facilities in combination with the resale of the telecommunications services of another carrier." The Commission concluded in the Ameritech Michigan Order that, when a BOC relies upon more than one competing provider to satisfy section 271(c)(1)(A), each carrier need not provide service to both residential and business customers. B. Discussion 62. We conclude that Bell Atlantic demonstrates that it satisfies the requirements of Track A based on the interconnection agreements it has implemented with competing carriers in New York. Specifically, we find that AT&T, MCI WorldCom, and Cablevision Lightpath provide telephone exchange service either exclusively or predominantly over their own facilities to residential subscribers and to business subscribers. The New York Commission also concludes that Bell Atlantic has met the requirements of section 271(c)(1)(A). None of the commenting parties, including the competitors cited by Bell Atlantic in support of its showing, challenge Bell Atlantic's assertion in this regard. Thus, Bell Atlantic meets the requirements of section 271(c)(1)(A). V. COMPLIANCE WITH CHECKLIST A. Checklist Item 1 - Interconnection 1. Non-Pricing Aspects of Interconnection a. Background 63. Section 271(c)(2)(B)(i) of the Act requires a section 271 applicant to provide "[i]nterconnection in accordance with the requirements of sections 251(c)(2) and 252(d)(1)." Section 251(c)(2) imposes a duty on incumbent LECs "to provide, for the facilities and equipment of any requesting telecommunications carrier, interconnection with the local exchange carrier's network . . . for the transmission and routing of telephone exchange service and exchange access." In the Local Competition First Report and Order, the Commission concluded that interconnection referred "only to the physical linking of two networks for the mutual exchange of traffic." Section 251 contains three requirements for the provision of interconnection. First, an incumbent LEC must provide interconnection "at any technically feasible point within the carrier's network." Second, an incumbent LEC must provide interconnection that is "at least equal in quality to that provided by the local exchange carrier to itself." Finally, the incumbent LEC must provide interconnection "on rates, terms, and conditions that are just, reasonable, and nondiscriminatory, in accordance with the terms of the agreement and the requirements of [section 251] and section 252." 64. To implement the equal-in-quality requirement in section 251, the Commission's rules require an incumbent LEC to design and operate its interconnection facilities to meet "the same technical criteria and service standards" that are used for the interoffice trunks within the incumbent LEC's network. In the Local Competition First Report and Order, the Commission identified trunk group blockage and transmission standards as indicators of an incumbent LEC's technical criteria and service standards. In prior section 271 applications, the Commission concluded that disparities in trunk group blockage indicated a failure to provide interconnection to competing carriers equal-in-quality to the interconnection the BOC provided to its own retail operations. 65. In the Local Competition First Report and Order, the Commission concluded that the requirement to provide interconnection on terms and conditions that are "just, reasonable, and nondiscriminatory" means that an incumbent LEC must provide interconnection to a competitor in a manner no less efficient than the way in which the incumbent LEC provides the comparable function to its own retail operations. The Commission's rules interpret this obligation to include, among other things, the incumbent LEC's installation time for interconnection service and its provisioning of two-way trunking arrangements. Similarly, repair time for troubles affecting interconnection trunks is useful for determining whether a BOC provides interconnection service under "terms and conditions that are no less favorable than the terms and conditions" the BOC provides to its own retail operations. 66. Competing carriers may also choose any method of technically feasible interconnection at a particular point on the incumbent LEC's network. Incumbent LEC provision of interconnection trunking is one common means of interconnection. Technically feasible methods also include, but are not limited to, physical and virtual collocation and meet point arrangements. In the Advanced Services First Report and Order, the Commission revised its collocation rules to require incumbent LECs to include shared cage and cageless collocation arrangements as part of their physical collocation offerings. The provision of collocation is an essential prerequisite to demonstrating compliance with item 1 of the competitive checklist. To show compliance with its collocation obligations, a BOC must have processes and procedures in place to ensure that all applicable collocation arrangements are available on terms and conditions that are "just, reasonable, and nondiscriminatory" in accordance with section 251(c)(6) and our implementing rules. Data showing the quality of procedures for processing applications for collocation space, as well as the timeliness and efficiency of provisioning collocation space, helps the Commission evaluate a BOC's compliance with its collocation obligations. b. Discussion 67. We are persuaded, for the reasons discussed below, that Bell Atlantic demonstrates that, in New York, it provides equal-in-quality interconnection on terms and conditions that are just, reasonable, and nondiscriminatory in accordance with the requirements of section 251(c)(2) and 252(d)(1), as specified in section 271. We further find that Bell Atlantic meets its burden of proof that it designs its interconnection facilities to meet "the same technical criteria and service standards" that are used for the interoffice trunks within its own network, and that Bell Atlantic makes interconnection available at any technically feasible point. Finally, we find that Bell Atlantic demonstrates that it is providing collocation in New York in accordance with the Commission's rules. (i) Interconnection Trunking 68. Based on our review of the record, we are persuaded that Bell Atlantic provides competing carriers with interconnection trunking in New York that is equal-in-quality to the interconnection Bell Atlantic provides to its own retail operations, and on terms and conditions that are just, reasonable, and nondiscriminatory. Bell Atlantic makes interconnection available in New York through interconnection agreements and through a state approved tariff. Bell Atlantic receives orders for interconnection trunks through the Access Service Request (ASR) process, and accepts ASRs through an electronic application-to-application interface, its Internet Web Graphical User Interface (GUI), and manual orders. In addition, Bell Atlantic provides performance data to measure the quality of interconnection service provided to competing carriers. 69. In prior section 271 applications, we relied heavily on trunk group blockage data to evaluate a BOC's interconnection quality. Bell Atlantic's performance data show that, in the months leading up to its application, Bell Atlantic provided interconnection using the level of service that is received in its own network. Specifically, Bell Atlantic's performance data show that, for the three months immediately preceding its section 271 application, interconnection trunk groups provided to competing carriers experienced blockage less frequently than Bell Atlantic's own retail trunk groups. The comments of the New York Commission, Intermedia, and Nextlink corroborate Bell Atlantic's performance data, and further indicate that Bell Atlantic provides interconnection equal-in-quality to the interconnection provided to Bell Atlantic's own retail operations. As a final matter, we note that the failure of any commenter to raise trunk group blockage as an issue further supports our conclusion that Bell Atlantic adequately designs its interconnection facilities to ensure calls are completed. 70. We find that other aspects of Bell Atlantic's data further indicate that Bell Atlantic is providing nondiscriminatory interconnection trunking in New York. Bell Atlantic's performance data show that, for July and August 1999, Bell Atlantic rarely missed installation appointments for provisioning interconnection trunks for competitors. In fact, Bell Atlantic missed installation appointments for local exchange competitors less often than it did for interexchange carriers in July and August, and we note that Bell Atlantic's data show that Bell Atlantic provided comparable installation quality through September. 71. We have examined the issues pointed out by the Department of Justice, Teligent, e.spire, Allegiance, and others regarding Bell Atlantic's provisioning of new and large orders of interconnection trunks. These parties generally argue that requesting carriers have experienced unreasonable delays in Bell Atlantic provisioning of new and large orders of interconnection trunks. In its application, Bell Atlantic submitted performance data that showed a statistically significant difference between the provisioning of trunks for competitive LECs and for interexchange carriers as reflected in some performance measurements related to provisioning large orders of interconnection trunks. After further analysis and discussion with the Commission, Bell Atlantic identified significant errors in its New York Carrier-to-Carrier Performance Reports, and submitted revised data. In addition, Bell Atlantic submitted supplementary data to show its provisioning performance for interconnection trunks provided to both competitive LECs and interexchange carriers. Our review of Bell Atlantic's supplementary data shows that, although its provisioning performance has deteriorated since January 1999, Bell Atlantic's provisioning of interconnection trunks for competitive LECs is comparable to its performance for interexchange carriers, which indicates that Bell Atlantic is meeting its equal-in- quality obligations. We therefore conclude that, while the claims of e.spire and others may very well be true, evidence of such provisioning delays does not preclude a showing of compliance for section 271 purposes, so long as the equal-in-quality requirement is met. 72. We conclude that our decision that Bell Atlantic meets checklist item 1 rests upon its demonstration that trunk group blockage for competitors is lower than for Bell Atlantic's retail operations, Bell Atlantic's rate of missed installation appointments is lower for service to local competitors than for service to interexchange carriers, and there is no significant difference between its provisioning of interconnection trunks to local competitors and to interexchange carriers. For the benefit of future section 271 applications, and for purposes of evaluating Bell Atlantic's continued compliance with section 271(c)(2)(B)(I), we emphasize that our conclusion is based on a weighing of the various factors discussed in the foregoing paragraphs. A different combination of factors in another case might well lead us to conclude that, on the whole, competitive LECs do not receive equal-in-quality interconnection on just, reasonable, and nondiscriminatory terms and conditions. (ii) Collocation 73. Bell Atlantic has demonstrated that its collocation offering in New York satisfies the requirements of sections 271 and 251 of the Act. Bell Atlantic provides physical and virtual collocation through a state-approved tariff. In its application, Bell Atlantic indicates that shared, cageless, and adjacent collocation options are available in New York, and that it has taken other steps to implement the collocation requirements contained in the Advanced Services First Report and Order. In addition, Bell Atlantic demonstrates that it has deployed methods and procedures designed to ensure that its business units implement the Commission's collocation rules, including the designation of employees dedicated to providing collocation to competitive LECs, standard operating procedures related to collocation, and its CLEC HANDBOOK, which informs collocators of their rights and responsibilities. A number of commenters, including the New York Commission and several competitive LECs, agree with Bell Atlantic that its collocation offerings have been revised to reflect the requirements specified in the Advanced Services First Report and Order. 74. We disagree with the contentions of ALTS that the New York state tariff, and the New York Commission tariff review process, do not adequately ensure that Bell Atlantic's collocation offerings are consistent with section 251 and the Commission's rules. Specifically, ALTS contends that terms in the New York state tariff delay the provisioning of collocation space and impose restrictions on methods of interconnection and access to collocation. In addition, ALTS argues that the New York tariff does not clarify Bell Atlantic's allocation of collocation costs. After reviewing the record, we are persuaded by the New York Commission that Bell Atlantic is meeting its collocation obligations. Bell Atlantic revised its tariffed collocation offering to make it consistent with our Advanced Services First Report and Order. Bell Atlantic's collocation tariff underwent an active and thorough review at the state level. The New York Commission addressed the provisioning of collocation space and established standard provisioning intervals for caged, cageless, and virtual collocation. 75. Our review of Bell Atlantic's collocation performance data indicates that Bell Atlantic responds to applications for collocation space in a timely manner. Between May 1999 and August 1999, Bell Atlantic processed 667 requests for collocation space and almost always responded to such requests within the 8-day standard set by the New York Commission. Although we are concerned that Bell Atlantic's performance data shows recent failures to meet the 76-day provisioning interval established by the New York Commission for physical collocation, our finding of checklist compliance is predicated on Bell Atlantic's demonstration that 95% of the time it provisions collocation within the 76-day provisioning interval established by the New York Commission. Should these recent failures lead to a more widespread deterioration in provisioning collocation, however, enforcement action pursuant to section 271(d)(6) may be appropriate. (iii) Technically Feasible Points of Interconnection 76. We conclude that Bell Atlantic provides interconnection at all technically feasible points, as required by our rules, and therefore demonstrates checklist compliance. Bell Atlantic asserts that it makes interconnection available at all technically feasible points, including trunk- side at Bell Atlantic end offices and access tandems and line-side at Bell Atlantic end offices. Bell Atlantic demonstrates that it has an approved state tariff that spells out readily available points of interconnection, and provides a process for requesting interconnection at additional, technically-feasible points. We disagree with Sprint that its experience negotiating interconnection agreements with Bell Atlantic conclusively demonstrates that Bell Atlantic has violated its obligation to permit competing carriers to select interconnection points. Sprint's experience does not constitute evidence of systematic failures by Bell Atlantic to provide interconnection at all technically feasible points. Bell Atlantic points out that a state-approved process enables competitive LECs to obtain interconnection at technically feasible points not specified in the tariff, and the comments of the New York Commission support this statement. We agree with the New York Commission that the pending arbitration between Sprint and Bell Atlantic is the appropriate forum for addressing this issue. As a final matter, we conclude that Bell Atlantic has demonstrated that it provides two-way trunking in accordance with our rules, and no commenter presents credible information to show otherwise. 2. Pricing of Collocation a. Background 77. In order to comply with its collocation obligations, a BOC must make physical and virtual collocation arrangements available at rates that are "just, reasonable, and nondiscriminatory" in accordance with section 251(c)(6) of the Act and our rules implementing that section. Although the Commission's pricing rules were stayed by the U.S. Court of Appeals for the Eighth Circuit in 1996, pricing authority was restored by the Supreme Court on January 25, 1999. In reaching its decision, the Court acknowledged that section 201(b) "explicitly grants the FCC jurisdiction to make rules governing matters to which the 1996 Act applies." Furthermore, the Court determined that section 251(d) also provides evidence of an express jurisdictional grant by requiring that "the Commission [shall] complete all actions necessary to establish regulations to implement the requirements of this section." The Court also held that the pricing provisions implemented under the Commission's rulemaking authority do not inhibit the establishment of rates by the States. The Court concluded that the Commission has jurisdiction to design a pricing methodology to facilitate local competition under the 1996 Act, including pricing for interconnection and unbundled access, as "it is the States that will apply those standards and implement that methodology, determining the concrete result." b. Discussion 78. Based on the evidence in the record, we find that Bell Atlantic offers cageless physical collocation to those LECs that request it at just, reasonable, and nondiscriminatory prices, in compliance with checklist item 1. Commenters raised only two issues related to collocation prices, and, as discussed below, we find that these commenters misinterpreted Bell Atlantic's tariffs and their concerns are unfounded. Bell Atlantic asserts that its collocation prices are consistent with the Act and Commission rules. The New York Commission concludes that Bell Atlantic currently provides collocation under approved interconnection agreements and tariffs, consistent with FCC and New York Commission orders. We agree with the New York Commission that the issues raised by commenters with respect to checklist item 1 "do not preclude a finding that Bell Atlantic-NY is in compliance with this checklist item." The Department of Justice did not comment on Bell Atlantic's collocation prices. 79. We disagree with TRA's assertion that Bell Atlantic's collocation prices are discriminatory because they burden competing carriers with "unnecessary security measures and costs." These rates are not discriminatory because Bell Atlantic does not impose the costs of security measures. In Phase Three of its network elements rate case, the New York Commission held that Bell Atlantic may not recover any costs for cageless collocation security measures. Rather, it held that Bell Atlantic must bear such costs itself. Bell Atlantic later filed cageless security rates with the New York Commission, but these rates have not yet been approved and are not in effect. Despite the fact that competitors complained about the lack of set rates for cageless collocation security measures, the New York Commission did not impose temporary rates for cageless collocation security measures, holding instead that this cost and Bell Atlantic's associated cost justification will be considered in Phase Four of the New York Commission's unbundled network elements rate case. In its reply comments to Bell Atlantic's 271 application proceeding, the New York Commission noted that Bell Atlantic had a "placeholder" in its cageless collocation tariff for its security rate but that no rates are being imposed. We therefore find that TRA has misinterpreted Bell Atlantic's tariff and that its claim that Bell Atlantic's security rates are discriminatory is unfounded. 80. We also disagree with ALTS' claim that Bell Atlantic does not meet the Commission's requirements that it allocate its space preparation and related up-front costs among competing carriers on a pro-rata basis. In order to fulfill its obligation to provide nondiscriminatory access to interconnection, an incumbent LEC must "allocate space preparation, security measures, and other collocation charges on a pro-rated basis so the first collocator in a particular incumbent premises will not be responsible for the entire cost of site preparation." The New York Commission reviewed Bell Atlantic's interconnection tariff and rejected Bell Atlantic's initial proposal that it be allowed to charge the initial collocator the entire cost of space preparation. The New York Commission held that "it seems unreasonable to require the initial collocator to bear, up-front, the entire cost of protecting [Bell Atlantic] against the possibility that its costs may go unrecovered." The New York Commission further held that no reason existed to single out these costs for up-front recovery. The New York Commission instead estimated room construction costs and other up-front payments on a TELRIC basis and provided for their recovery through recurring charges. The New York Commission calculated on the basis of reasonable estimates of the likely number of users, thereby "obviating any possibility that the full cost would be imposed on the first [competing carrier]." Bell Atlantic has complied with this requirement in its tariff. Based on the record presented to us, we find that the New York Commission has set prices for a competing carriers' up-front site preparation costs at TELRIC- based costs, and ensured that the initial competitor to collocate will not bear the complete up- front collocation costs. Therefore, we conclude that this claim is without merit. B. Checklist Item 2 - Unbundled Network Elements 81. The nondiscriminatory provision of operations support systems (OSS) and the ability of competing carriers to combine unbundled network elements are integral aspects of the BOC's obligation to provide access to unbundled network elements as required by checklist item 2. In this section, we first outline section 271's nondiscrimination standard and our general approach to analyzing the adequacy of Bell Atlantic's OSS. We then briefly describe the critically important independent third-party testing conducted by KPMG and Hewlett Packard under the supervision of the New York Commission. Next, we describe briefly the systems, databases, and personnel on which Bell Atlantic relies in support of its claim that it provides access to OSS on a nondiscriminatory basis. We then address Bell Atlantic's change management process and the technical assistance that Bell Atlantic offers to competing carriers seeking to use its OSS. We also analyze Bell Atlantic's provision of access to the critical OSS functions of pre-ordering, ordering, provisioning, maintenance and repair, and billing. Finally, we analyze in this section whether Bell Atlantic provides access to unbundled network elements in a manner that allows competing carriers to combine such elements. 1. Operations Support Systems 82. As discussed below, we conclude that Bell Atlantic demonstrates that it provides requesting carriers nondiscriminatory access to OSS functions. Specifically, we find that Bell Atlantic provides a change management process and technical assistance that affords competing carriers a meaningful opportunity to compete. We also find that Bell Atlantic offers nondiscriminatory access to its pre-ordering, ordering, provisioning, maintenance and repair, and billing OSS functions. In reaching these conclusions, we acknowledge that we differ from the evaluation of the Department of Justice in certain material respects. Although we have accorded substantial weight to the Department's views as required by section 271, the statute prohibits us from giving the Department's views preclusive weight. With respect to access to OSS functions, we differ from the Department primarily in instances where we assess the totality of the evidence differently or where we have a greater amount of information available to inform our conclusions. a. Background 83. Incumbent LECs use a variety of systems, databases, and personnel (collectively referred to as OSS) to provide service to their customers. The Commission consistently has found that nondiscriminatory access to OSS is a prerequisite to the development of meaningful local competition. For example, new entrants must have access to the functions performed by the incumbent's OSS in order to formulate and place orders for network elements or resale services, to install service to their customers, to maintain and repair network facilities, and to bill customers. The Commission has determined that without nondiscriminatory access to the BOC's OSS, a competing carrier "will be severely disadvantaged, if not precluded altogether, from fairly competing" in the local exchange market. 84. Section 271 requires the Commission to determine whether a BOC offers nondiscriminatory access to OSS functions. Section 271(c)(2)(B)(ii) requires a BOC to provide "nondiscriminatory access to network elements in accordance with the requirements of sections 251(c)(3) and 252(d)(1)." The Commission has determined that access to OSS functions falls squarely within an incumbent LEC's duty under section 251(c)(3) to provide unbundled network elements under terms and conditions that are nondiscriminatory and just and reasonable, and its duty under section 251(c)(4) to offer resale services without imposing any limitations or conditions that are discriminatory or unreasonable. The Commission must therefore examine a BOC's OSS performance to evaluate compliance with section 271(c)(2)(B)(ii) and (xiv). In addition, the Commission has also concluded that the duty to provide nondiscriminatory access to OSS functions is embodied in other terms of the competitive checklist as well. Consistent with prior orders, we examine Bell Atlantic's OSS performance directly under checklist items 2 and 14, as well as other checklist terms. 85. As part of its statutory obligation to provide nondiscriminatory access to OSS functions, a BOC must provide access that sufficiently supports each of the three modes of competitive entry envisioned by the 1996 Act - competitor-owned facilities, unbundled network elements, and resale. For OSS functions that are analogous to those that a BOC provides to itself, its customers or its affiliates, the nondiscrimination standard requires the BOC to offer requesting carriers access that is equivalent in terms of quality, accuracy, and timeliness. The BOC must provide access that permits competing carriers to perform these functions in "substantially the same time and manner" as the BOC. The Commission has recognized in prior orders that there may be situations in which a BOC contends that, although equivalent access has not been achieved for an analogous function, the access that it provides is nonetheless nondiscriminatory within the meaning of the statute. 86. For OSS functions that have no retail analogue, the BOC must offer access "sufficient to allow an efficient competitor a meaningful opportunity to compete." In assessing whether the quality of access affords an efficient competitor a meaningful opportunity to compete, we will examine, in the first instance, whether specific performance standards exist for those functions. In particular, we will consider whether appropriate standards for measuring OSS performance have been adopted by the relevant state commission or agreed upon by the BOC in an interconnection agreement or during the implementation of such an agreement. If such performance standards exist, we will evaluate whether the BOC's performance is sufficient to allow an efficient competitor a meaningful opportunity to compete. 87. We analyze whether Bell Atlantic has met the nondiscrimination standard for each OSS function using the two-step approach outlined in prior orders. First, we determine "whether the BOC has deployed the necessary systems and personnel to provide sufficient access to each of the necessary OSS functions and whether the BOC is adequately assisting competing carriers to understand how to implement and use all of the OSS functions available to them." We next assess "whether the OSS functions that the BOC has deployed are operationally ready, as a practical matter." 88. Under the first inquiry, a BOC must demonstrate that it has developed sufficient electronic (for functions that the BOC accesses electronically) and manual interfaces to allow competing carriers equivalent access to all of the necessary OSS functions. For example, a BOC must provide competing carriers with the specifications necessary for carriers to design or modify their systems in a manner that will enable them to communicate with the BOC's systems and any relevant interfaces. In addition, a BOC must disclose to competing carriers any internal business rules and other formatting information necessary to ensure that a carrier's requests and orders are processed efficiently. Finally, a BOC must demonstrate that its OSS is designed to accommodate both current demand and projected demand for competing carriers' access to OSS functions. Although not a prerequisite, the Commission continues to encourage the use of industry standards as an appropriate means of meeting the needs of a competitive local exchange market. 89. Under the second inquiry, we examine performance measurements and other evidence of commercial readiness to ascertain whether the BOC's OSS is handling current demand and will be able to handle reasonably foreseeable demand volumes. The most probative evidence that OSS functions are operationally ready is actual commercial usage. Absent data on commercial usage, the Commission will consider the results of carrier-to-carrier testing, independent third-party testing, and internal testing in assessing the commercial readiness of a BOC's OSS. We reiterate, however, that the persuasiveness of a third-party review is dependent upon the qualifications, experience and independence of the third party and the conditions and scope of the review itself. b. Overview of OSS Operations 90. Bell Atlantic utilizes a number of systems and processes to support the entry of competing carriers into the local services market in New York. As an initial matter, a new entrant seeking to compete in the New York local services market must establish some form of connectivity with Bell Atlantic to submit service requests and receive responses. Bell Atlantic provides requesting carriers an application-to-application interface based on the Electronic Data Interchange (EDI) protocol for pre-ordering and ordering functions, as well as a Web-based Graphical User Interface (Web GUI or GUI) for pre-ordering, ordering and maintenance and repair functions. In addition, Bell Atlantic provides requesting carriers with training and reference guides for the use of each interface. A new entrant seeking to use the EDI interface must undergo a certification test with Bell Atlantic to verify that the carrier's operations support systems are capable of submitting valid service orders and receiving responses. 91. Before placing an actual order for service, a competing carrier can obtain pre- ordering information by sending a request over the Web GUI or EDI pre-ordering interface. Such pre-ordering information, which is often accessed while the customer is on the line, typically includes a customer's address and service history and the services and features available to that customer, as well as telephone numbers and delivery dates available from Bell Atlantic. Bell Atlantic returns the requested information over the same interface used by the carrier to submit the inquiry. The EDI interface enables competing carriers to populate an order form with information received from pre-ordering inquiries. 92. Using the information obtained in the pre-ordering process, the competing carrier submits an order for service using the EDI or Web GUI interface. An order sent by a competing carrier enters the Direct Customer Access System (DCAS) gateway system, which performs an initial check of the validity of the order. If the order is missing information or is determined not to be a valid transaction, Bell Atlantic will stop processing the order and send a Local Service Request Rejection (order rejection) notice to the carrier. An order that is not rejected will either flow automatically from DCAS to the Direct Order Entry (DOE) system or drop out for manual processing at a Telecom Industry Services Ordering Center (TISOC). At the TISOC, a Bell Atlantic representative will input the order into the Service Order Processor (SOP) directly. If the order flowed through to DOE, the order will pass through another series of checks and edits before it is passed to SOP for processing in the appropriate back end system. If the order does not pass the DOE screening, it is manually input into SOP by a Bell Atlantic representative. Once an order reaches SOP, it is mixed in and processed along with Bell Atlantic retail orders, and Bell Atlantic returns a Local Services Request Confirmation (order confirmation) to the carrier. The order confirmation provides, at minimum, the scheduled due date, service order identification, and account telephone number. At times, a carrier may need to "supplement" the order to reflect a subsequent change or to respond to an error message. 93. After an order is successfully entered into SOP, Bell Atlantic begins the process of provisioning the order, or activating the requested service or feature, which may involve assigning facilities, updating translations in a switch, and dispatching technicians. Specifically, an order flows from SOP to the Service Order Analysis and Control (SOAC) system. SOAC controls the progress of service orders through the provisioning process by distributing the service order to other necessary provisioning systems and then updating SOP. From SOAC, most orders flow automatically through the assignment systems, including the Loop Facility Assignment and Control System (LFACS), where the appropriate facilities are assigned or reserved for the order. After assignment, the next stage in the provisioning process for most orders is the loading of the translations into the switch, which is performed by the Recent Change Memory Administration Center (RCMAC). In addition, technicians at the central office perform any wiring work associated with the order. Orders that require work performed outside the central office are sent to the Work Force Administration (WFA) system for dispatch of a field technician. The Regional CLEC Coordination Center (RCCC) facilitates and coordinates the provisioning of wholesale orders. Competing carriers can monitor the provisioning process by viewing Bell Atlantic's regular posting of orders that are in jeopardy of missing an installation due date and by querying the order's status in SOP. Upon completion of the work involved in activating service, Bell Atlantic sends a notice of "work completion" to the carrier. In addition, after the order moves from SOP into Bell Atlantic's billing systems and is recorded as complete in the billing systems, Bell Atlantic sends a notice of "billing completion" to the carrier. 94. If a competing carrier's customer experiences service disruptions, the carrier can create and monitor trouble tickets, access trouble history for that line, and request a test of the customer's circuit by submitting inquiries over the Web GUI. A carrier's maintenance and repair inquiry is sent to the Repair Trouble Administration System (RETAS) gateway system, which routes requests to the appropriate back end systems and returns electronic responses. Most trouble reports are processed through the Loop Maintenance Operating System, handling overall maintenance, tracking and dispatch activities, and the StarMem system, which allows automatic feature updates to switches. To test for and analyze faults on a circuit, Bell Atlantic uses the Mechanized Loop Testing (MLT), Switched Access Remote Testing System (SARTS), and Delphi systems. Bell Atlantic's Regional CLEC Maintenance Center (RCMC) supports wholesale trouble reporting and repair issues. Bell Atlantic returns responses to trouble ticket inquiries over the same interface used by the carrier to submit the inquiry. 95. In order for competing carriers to bill their customers, Bell Atlantic provides carriers with usage billing information and a process for adjusting or correcting invalid or incorrect data. Bell Atlantic also provides requesting carriers documentation on its billing procedures, bill content and related interactions. Specifically, Bell Atlantic delivers a record of daily usage to competing carriers. Bell Atlantic also produces periodic bills (up to ten monthly) for wholesale carriers using the Customer Record Information System (CRIS), which provides billing for resale and unbundled loops, and the Customer Access Billing System (CABS), which provides billing for access services and other unbundled network elements. Competing carriers receive aggregated bills for the charges incurred by all their customers in a particular area, as well as charges for products and services ordered by the carrier itself. If a competing carrier believes that an individual usage item contains errors, it initiates a billing usage claim, and may be required to transmit the erroneous usage back to Bell Atlantic. Incorrect usage data may be either reprocessed or corrected with a billing adjustment. The competing carrier is responsible for billing the end user. c. Independent Third-Party Testing 96. The New York Commission retained KPMG to conduct an independent, third- party test of the readiness of Bell Atlantic's OSS, interfaces, documentation and processes. Over the course of fifteen months, KPMG evaluated 855 separate items relating to pre-ordering, ordering, provisioning, maintenance and repair, billing, and relationship management and infrastructure, by performing both transaction and operational tests. KPMG combined efforts with Hewlett Packard to accomplish the transaction-driven tests. In doing so, KPMG acted much like a "pseudo-competing carrier" operations department, working with Bell Atlantic business rules, creating and tracking orders, monitoring Bell Atlantic performance, logging trouble tickets, and evaluating carrier-to-carrier bills. At the same time, Hewlett Packard acted as a competing carrier information technology department, establishing electronic bonding with Bell Atlantic, translating back and forth between business and EDI rule formats, and resolving problems with missing orders and responses. By building and submitting transactions using Bell Atlantic's electronic interfaces with test accounts in central offices spread across New York, KPMG was able to live the experience of a competing carrier. In addition, KPMG used operational tests to evaluate the results of Bell Atlantic day-to-day operational management and change management processes to determine if they functioned in accordance with Bell Atlantic documentation and expectations. 97. KPMG's test was broad in scope. All stages of the relationship between Bell Atlantic and competing carriers were considered, from establishing the initial relationship, to performing daily operations, to maintaining the relationship. Resale, UNE-loops, UNE- platform, and combinations were all included in the test. In addition, both the application-to- application electronic data interchange (EDI) and the terminal-type web-based graphical user interface (GUI) were tested. KPMG performed pre-ordering, ordering, provisioning, maintenance and repair, billing, and relationship management and infrastructure tests to evaluate functional capabilities and determine whether competing carriers receive a level of service comparable to Bell Atlantic retail service. To fully test these systems, orders were submitted with known error conditions, canceled, and supplemented. Documentation was evaluated for usefulness, correctness, and completeness. KPMG also performed stress volume tests of Bell Atlantic systems and identified specific bottlenecks for wholesale customers. 98. In performing these tests, KPMG adopted a military-style test philosophy, or a mindset of "test until you pass." Thus, when situations arose where testing revealed that a Bell Atlantic process, document, or system did not meet expectations, Bell Atlantic would generally implement a fix and KPMG would retest the process, document, or system until satisfied. As a result, KPMG believes that competing carriers now have a "baseline set of working components" that a one-time diagnostic evaluation of Bell Atlantic's OSS would not have provided. 99. To the greatest extent possible, the KPMG test was both independent and blind. Neither KPMG nor Hewlett Packard had a reporting relationship to Bell Atlantic. Although it was virtually impossible for the KPMG transactions to be truly blind, KPMG instituted certain procedures to ensure that both KPMG and Hewlett Packard would not receive preferential treatment. For example, KPMG required that all documents provided to them were generally available to all competing carriers. The New York Commission monitored phone calls between KPMG and Hewlett Packard and Bell Atlantic, and competing carriers were invited to attend conference calls. In addition, KPMG made concurrent observations of the service quality delivered to other competing carriers during the course of its test. 100. The scope and depth of KPMG's review, and the conditions surrounding it, including KPMG's independence, military-style test philosophy, efforts to place themselves in the position of an actual market entrant, and efforts to maintain blindness when possible, lead us to treat the conclusions in the KPMG Final Report as persuasive evidence of Bell Atlantic's OSS readiness. As we have said before, the persuasiveness of a third-party review is dependent on the conditions and scope of the review. Because we recognize that various third-party tests may be adequate to demonstrate the operational readiness of a BOC's OSS, we emphasize that we do not foreclose the possibility that a third-party test designed differently than the KPMG review may also be persuasive. Nonetheless, were a third-party test less comprehensive, less independent, less blind, and, therefore, less useful in assessing the real world impact of a BOC's OSS on competing carriers, we would not necessarily find it persuasive and may accord it less weight than we do the KPMG Final Report. d. Change Management and Technical Assistance (i) Change Management 101. We conclude that Bell Atlantic demonstrates that it provides the documentation and support necessary to give competing carriers nondiscriminatory access to its OSS. Bell Atlantic makes this demonstration by showing that it has an adequate change management process in place in New York. The record also reflects that Bell Atlantic has adhered to its change management process over time. As a result, we find that Bell Atlantic provides access to its OSS in a manner that allows an efficient competitor a meaningful opportunity to compete. (a) Background 102. Competing carriers need information about and specifications for an incumbent's systems and interfaces in order to develop and modify their systems and procedures to access the incumbent's OSS functions. Thus, in the Ameritech Michigan Order, the Commission determined that in order to provide nondiscriminatory access to OSS, a BOC must first demonstrate that it "has deployed the necessary systems and personnel to provide sufficient access to each of the necessary OSS functions and . . . is adequately assisting competing carriers to understand how to implement and use all of the OSS functions available to them." By showing that it adequately assists competing carriers to use available OSS functions, a BOC provides evidence that it offers an efficient competitor a meaningful opportunity to compete. As part of this demonstration, the Commission will give substantial consideration to the existence of an adequate change management process and evidence that the BOC has adhered to this process over time. 103. The change management process refers to the methods and procedures that the BOC employs to communicate with competing carriers regarding the performance of and changes in the BOC's OSS system. Such changes may include operations updates to existing functions that impact competing carrier interface(s) upon a BOC's release of new interface software; technology changes that require competing carriers to meet new technical requirements upon a BOC's software release date; additional functionality changes that may be used at the competing carrier's option, on or after a BOC's release date for new interface software; and changes that may be mandated by regulatory authorities. Without a change management process in place, a BOC can impose substantial costs on competing carriers simply by making changes to its systems and interfaces without providing adequate testing opportunities and accurate and timely notice and documentation of the changes. As Allegiance suggests, change management problems can impair a competing carrier's ability to obtain nondiscriminatory access to UNEs, and hence a BOC's compliance with section 271(c)(2)(B)(ii). 104. Competing carriers have had a substantial role in the development of Bell Atlantic's change management process in New York. As part of a collaborative process dating back to October 1997 and conducted under the auspices of the New York Commission, Bell Atlantic and competing carriers developed a detailed process of managing changes to the Bell Atlantic systems and interfaces that affect competing carriers. This process resulted in the May 1998 document entitled "Telecom Industry Services-Change Management Process" (Change Agreement). Although there have been subsequent modifications to the Change Agreement, the basic process and timelines set out in this document are still applicable. 105. The Change Agreement sets forth detailed procedures for introducing changes in Bell Atlantic's systems and documentation. It divides all changes into five different categories and provides specific time lines and intervals for each category. Thus, the process is designed to accommodate emergency changes, regulatory changes, changes in industry standards, changes requested by Bell Atlantic, and changes requested by competing carriers. 106. Regardless of the type of change, the Change Agreement expressly provides for feedback from competing carriers on the proposed changes. In addition, the Change Agreement calls for Bell Atlantic and the competing carriers to develop jointly a schedule for the distribution of draft specifications or business rules, receipt of competing carrier comments on the documentation, and distribution of final documentation. Bell Atlantic has established a forum where representatives from Bell Atlantic and competing carriers meet-often more than once a month-to discuss upcoming system and interface changes as well as the change management procedures themselves. Moreover, in September 1999, representatives of Bell Atlantic and competing carriers began to prioritize changes based on merit, rather than the sponsor of the change. Thus, competing carriers had a substantial role in the development of methods and procedures for the change management process in New York and continue to have opportunities for meaningful input in the change management process today. 107. Bell Atlantic's basic change management process is memorialized and set forth in a single document, the Change Agreement. As a result, Bell Atlantic's change management process documentation is clearly organized and readily accessible to competing carriers. Competing carriers can readily access the Change Agreement on Bell Atlantic's Telecommunications Industry Services (TIS) web page. Modifications to this document are also available on the TIS web page. Moreover, in response to KPMG findings, Bell Atlantic has improved its procedures for competing carriers to cross-reference and track information regarding the change management process. Thus, Bell Atlantic now updates and maintains a database that tracks the progress of each specified change, reports changes systematically using change request numbers and uses these same numbers in communications with competing carriers to identify specific changes. 108. Bell Atlantic's change management process includes a method for dispute resolution that is separate and apart from any process that is set forth in interconnection agreements. As a result, competing carriers now have a forum specifically designed to address any change management disputes. In response to concerns raised by competing carriers, Bell Atlantic, in consultation with competing carriers and the New York Commission staff, established an escalation process for resolving change control disputes. This process allows competing carriers to appeal to upper level management at Bell Atlantic on change management issues and also allows competing carriers to raise these issues before the New York Commission staff. 109. Bell Atlantic's change management process provides for a stable testing environment. Competing carriers need access to a stable testing environment to certify that their OSS will be capable of interacting smoothly and effectively with Bell Atlantic's OSS, as modified. In addition, prior to issuing a new software release or upgrade, the BOC must provide a testing environment that mirrors the production environment in order for competing carriers to test the new release. If competing carriers are not given the opportunity to test new releases in a stable environment prior to implementation, they may be unable to process orders accurately and unable to provision new customer services without delays. KPMG originally found Bell Atlantic's testing environment "Not Satisfied," specifically noting that the testing environment "did not adequately mirror production capabilities." As the New York Commission suggests, this can result in competing carriers' transactions succeeding in the testing environment but failing in production. 110. In response to KPMG's initial finding, Bell Atlantic worked with New York Commission staff and competing carriers to establish a new testing environment and new testing procedures. Some of these changes were introduced in April 1999 as part of an interim Quality Assurance (QA) environment for carrier-to-carrier testing of new versions of OSS interfaces. KPMG reviewed the interim QA testing environment for pre-ordering and ordering and determined that the interim environment mirrored the production environment. At the same time, KPMG determined the availability of the testing environment under Bell Atlantic's interim procedures presented problems for competing carriers. As AT&T and MCI WorldCom note, the interim QA testing environment was only made available to competing carriers during business hours and for a maximum period of five business days. On September 20, 1999 Bell Atlantic introduced its permanent QA testing environment. Bell Atlantic represents that the permanent QA testing environment mirrors production and provides a physically separate environment for competing carrier testing. In addition, Bell Atlantic plans to maintain this testing environment for all but emergency changes for at least a month, including extended daily hours. Moreover, in order to ensure that competing carriers are not forced to test and cut over to a new industry standard release prematurely, Bell Atlantic maintains a pre-existing version after issuing a major new release rather than switching directly from one version to the next. Finally, Bell Atlantic, in response to a separate KPMG "Not Satisfied" finding, has introduced new procedures to certify that a competing carrier may move from the testing environment to the production environment. (b) Discussion 111. Based on the above record evidence, we conclude that Bell Atlantic demonstrates that it has a change management process in place in New York that provides an efficient competitor with a meaningful opportunity to compete. Specifically, we find that Bell Atlantic makes this showing with: (1) evidence of competing carrier input in the design and continued operation of the change management process; (2) the memorialization of the change management process in a basic document; (3) the availability of a separate forum for change management disputes; (4) and the availability of a stable testing environment that mirrors production. We note that even competing carriers have acknowledged in their comments that the processes in the Change Agreement are satisfactory as written. Because we recognize that various change management plans may be adequate to meet the needs of competing carriers, we emphasize that the individual factors described above are indicative, but not dispositive, of an adequate process. Although we will look for evidence of these same factors in evaluating a future applicant's change management process, we do not foreclose the possibility that a different plan may be sufficient. 112. We also find that the record demonstrates that Bell Atlantic has adhered to its change management process over time. Commenters, however, express concern that problems remain with respect to Bell Atlantic's ability to adhere to notification and documentation timelines in its Change Agreement and Bell Atlantic's ability to show that the permanent QA testing environment meets the needs of competing carriers. In addition, commenters allege that Bell Atlantic issues too many emergency changes and fails to consider competing carrier input in the change management process. (i) Notification and Documentation Timeliness 113. We conclude that Bell Atlantic provides competing carriers with change management notification and documentation for upcoming change releases in a manner sufficiently timely to allow an efficient competitor a meaningful opportunity to compete. As TRA suggests, the failure of a BOC to provide timely, complete, and accurate notice of alterations to its systems and processes hinders the ability of competitive providers to serve their customers adequately. Without timely notification and documentation, competing carriers are unable to modify their existing systems and procedures or develop new systems to maintain access to a BOC's OSS functions. As a preliminary matter, we find that the Change Agreement establishes reasonable intervals for the distribution of change management notification and documentation because they provide competing carriers with sufficient time to prepare for Bell Atlantic system changes. In addition, we commend Bell Atlantic and the New York Commission for developing metrics that report its compliance with these intervals. 114. We find that Bell Atlantic provides competing carriers with timely change management notification and documentation for changes made at the request of regulatory authorities (Type 2 changes), industry standard organizations (Type 3 changes), and competing carriers (Type 5 changes) in a manner sufficiently timely to allow an efficient competitor a meaningful opportunity to compete. For these types of changes, the data are extremely limited because they occur infrequently. Nonetheless, the data provided on these changes in both the Carrier-to-Carrier metrics and the KPMG Final Report demonstrate that Bell Atlantic has already established a pattern of compliance with the relevant notification and documentation intervals in its Change Agreement. 115. We also find that Bell Atlantic provides competing carriers with notification and documentation for Bell Atlantic-initiated changes (Type 4 changes) in a manner sufficiently timely to allow an efficient competitor a meaningful opportunity to compete. In its Final Report, KPMG found that Bell Atlantic was unable to meet documentation intervals set in the Change Agreement for Type 4 changes, and characterized this problem as "Not Satisfied." KPMG found that Bell Atlantic provided timely documentation in only three of nineteen instances for Type 4 changes from January to June 1999. During the same period, Bell Atlantic was able to provide timely notification of upcoming Type 4 changes in sixteen of twenty instances. Bell Atlantic contends, however, that it has now addressed the documentation timeliness problem identified by KPMG. With respect to initial notification timeliness, during the period from July to October 1999, the record shows that Bell Atlantic provided timely notification for eleven of twelve Type 4 changes. With respect to final documentation timeliness, during the period from August to October 1999, the record shows that Bell Atlantic provided timely documentation for eight of ten Type 4 changes. Thus, Bell Atlantic has demonstrated considerable improvement since the KPMG review. In particular, Bell Atlantic was able to provide both timely notification and documentation to competing carriers for two of two Type 4 changes that occurred in October 1999. We find that these improvements, coupled with the opportunities competing carriers have to participate in the prioritization of changes and the month long testing opportunities provided for Type 4 changes, indicate that an efficient competitor has a meaningful opportunity to compete. 116. In addition, we conclude that Bell Atlantic provides notification for emergency changes (Type 1 changes) in a manner sufficiently timely to allow an efficient competitor a meaningful opportunity to compete. Under the Change Agreement, timely emergency notification simply requires notification prior to implementation. As the KPMG Final Report suggests, timely emergency notification can range from several hours to several days advance notice. Although we understand advance notification is preferable for competing carriers, we also must acknowledge that given the nature of emergency changes, it will not always be possible for Bell Atlantic to notify competing carriers prior to implementation. Some commenters question Bell Atlantic's ability to provide competing carriers with timely notification of Type 1 emergency changes. MCI WorldCom, for instance, complains that the timeliness of Bell Atlantic's emergency notification fell considerably in September 1999, when Bell Atlantic was timely for only seven of twelve Type 1 changes. We note, however, that Bell Atlantic's Type 1 change notification was timely for twenty-five of twenty-six changes in July 1999 and six of six changes that occurred between October 1 and October 19, 1999. Because we believe that as a matter of course emergency changes will occur in situations where Bell Atlantic may be unable to notify competing carriers prior to implementation, we do not find that Bell Atlantic's September 1999 performance prevents us from concluding that Bell Atlantic provides emergency change notification to competing carriers in a manner sufficiently timely to allow an efficient competitor to compete. 117. Our conclusion that Bell Atlantic provides timely change management notification and documentation to competing carriers seeking to use its OSS differs from that reached by the Department of Justice. We reach this conclusion, however, by separately assessing the underlying issues associated with each of the Bell Atlantic change types identified in the Change Agreement. First, with respect to the limited number of changes made at the request of regulatory authorities, industry standard organizations, and competing carriers themselves, Bell Atlantic has established a pattern of general compliance with the notification and documentation intervals in its Change Agreement. Second, we find the recent improvement in Bell Atlantic's timely distribution of Type 4 notification and documentation demonstrates its ability to adhere to its change management process. Finally, while we acknowledge notification prior to implementation of an emergency change will not always be possible, we still find that Bell Atlantic provides sufficiently timely notification to competing carriers. 118. Although we reach the same conclusion as the New York Commission with respect to Bell Atlantic's change management notification and documentation timeliness, we do not rely on Bell Atlantic's willingness to have its future change management notification and documentation timeliness enforced through the Change Control Assurance Plan. In addition, we acknowledge that the timeliness of Bell Atlantic's performance falls short of the monthly standards for change management notification and documentation set out in the Carrier-to-Carrier metrics and used in the Change Control Assurance Plan. Nonetheless, when we view Bell Atlantic's overall performance over the course of recent months, we find that Bell Atlantic's notification and documentation timeliness is sufficient to allow an efficient competitor a meaningful opportunity to compete. We will, however, be prepared to take appropriate enforcement action if there is evidence of deteriorating performance in the future. Finally, although our conclusion is based on the specific categories of changes identified in the Bell Atlantic Change Agreement in place in New York, we do not foreclose the possibility that a different plan with a less disaggregated structure and different intervals for notification and documentation may also be sufficient. (ii) Testing Environment 119. We conclude that Bell Atlantic's permanent QA testing environment provides competing carriers with a stable environment and an adequate opportunity to test Bell Atlantic OSS changes prior to implementation. Specifically, we find the record demonstrates that Bell Atlantic's new testing environment adequately mirrors the production environment and offers the extended testing periods that competing carriers need for new entrant certification and new release testing. MCI WorldCom and AT&T note that as of the date of Bell Atlantic's application, no competing carriers had been given the opportunity to use the permanent QA testing environment and determine that it works in the manner Bell Atlantic represents in its application. We conclude there is sufficient evidence to demonstrate that Bell Atlantic's permanent QA testing environment provides a stable testing environment for competing carriers. 120. We base this conclusion on the experience of the competing carriers that used the permanent QA testing environment without difficulty for an October 16, 1999 software release. Thus, we find that the recent evidence from commercial usage suggests that Bell Atlantic's permanent QA environment works in the manner represented in its application. As the New York Commission attests, with only one minor exception, the results of the production run matched the results of the run in the permanent QA testing environment. The one exception, the absence of a billing telephone number for a directory listing, has been corrected. 121. Our conclusion is buttressed by the similarity between the interim and permanent QA testing environment and KPMG's finding that the interim testing environment adequately mirrored the production environment. Both environments mirror production and offer test decks of representative pre-ordering and ordering transactions. The basic processes for new release and new entrant testing distributed in April 1999 apply to both the interim and permanent environments. The only differences between the two environments are that the permanent QA testing environment is physically separate and expands the test period to one month, thus remedying the major problems identified by KPMG and competing carriers with the interim QA testing environment. 122. We find that the record demonstrates that Bell Atlantic's permanent QA testing environment provides competing carriers with a stable environment and adequate opportunity to test Bell Atlantic OSS changes prior to implementation. Although we reach the same conclusion as the New York Commission, we differ somewhat from that reached by the Department of Justice. The Department of Justice found that while it was hopeful that the permanent QA testing environment would meet competing carrier needs, the results of recent Bell Atlantic improvements did not appear in the record before them. Comments filed subsequent to the evaluation of the Department of Justice, however, demonstrate that the October 16, 1999 software release using the new QA testing environment was successful. As a result, we find that the record now demonstrates that Bell Atlantic provides a testing environment for OSS changes sufficient to enable an efficient competitor a meaningful opportunity to compete. (iii) Other Issues 123. AT&T and Sprint assert that Bell Atlantic improperly categorizes a substantial number of changes as Type 1 emergency changes in order to evade the longer notification requirements associated with other types of changes under the Change Agreement. We conclude these claims do not warrant a finding that Bell Atlantic fails to adhere to its change management procedures in a manner sufficient to provide an efficient competitor with a meaningful opportunity to compete. Type 1 emergency changes are specifically defined and provided for in the Change Agreement that was developed in a collaborative proceeding involving Bell Atlantic, competing carriers, and the New York Commission. Furthermore, as AT&T itself acknowledges, on June 30, 1999, Bell Atlantic and competing carriers began a series of workshops that resulted in a more narrow definition of Type 1 changes. This provides evidence of competing carriers' continuing opportunity to provide meaningful input in the change management process in New York. Since these workshops began, Bell Atlantic has reduced the number of Type 1 changes from twenty-six in July 1999 to ten in August, twelve in September and six in the first half of October. Because emergency changes are specifically provided for in the Change Agreement and Bell Atlantic's use of them has decreased in recent months, we find AT&T and Sprint's claims unpersuasive. 124. AT&T and MCI WorldCom allege that Bell Atlantic fails to give competing carriers opportunities to provide input on new releases as it is obligated to do under the Change Agreement. We find that the record simply does not support this claim. For instance, representatives of competing carriers and Bell Atlantic jointly prioritize upcoming changes. In addition, Bell Atlantic and competing carriers meet regularly to discuss upcoming changes and the change management process itself. As part of these meetings, Bell Atlantic and the competing carriers develop a detailed chart of competing carrier requests for action on specific change management issues, track the status of these problems, and note Bell Atlantic actions taken to address the problem. For example, when MCI WorldCom expressed a preference regarding how customer service record addresses be made available to competing carriers, Bell Atlantic agreed to add this functionality within the remaining weeks before the related change release. At the same time, Bell Atlantic devised a special software approach to defer implementation of this functionality for AT&T, the sole competing carrier that objected to this change. Although we would be concerned about the impact of a BOC disregarding input from competing carriers on change management issues, we do not believe the record indicates that this is a problem for carriers working with Bell Atlantic in New York. 125. We also conclude that problems with specific OSS changes described by MCI WorldCom, Allegiance, and Sprint do not warrant a conclusion that Bell Atlantic fails to adequately assist competing carriers seeking to use its OSS. Because Bell Atlantic must accommodate a variety of interests with any given change release, we reasonably expect some competing carriers to be less than satisfied with any given change. We do not, however, find that these complaints evidence a systemic problem. (ii) Technical Assistance and Help Desk Support 126. In the Ameritech Michigan Order, the Commission determined that in order to provide nondiscriminatory access to OSS, a BOC must first demonstrate that it "has deployed the necessary systems and personnel to provide sufficient access to each of the necessary OSS functions and . . . is adequately assisting competing carriers to understand how to implement and use all of the OSS functions available to them." By showing that it adequately assists competing carriers to use available OSS functions, a BOC provides evidence that it offers an efficient competitor a meaningful opportunity to compete. As part of this demonstration, the Commission will give substantial consideration to evidence showing that the BOC provides adequate technical assistance and help desk support to competing carriers seeking to use its OSS. 127. We conclude that Bell Atlantic demonstrates that it provides the technical assistance and help desk support necessary to give competing carriers nondiscriminatory access to its OSS. Bell Atlantic has produced a separate three volume handbook for resellers and purchasers of UNEs, both available on CD-ROM with word search capability. Documentation is updated for each release and also is made available on Bell Atlantic's web site. Thus, competing carriers have access to complete, up-to-date business rules and ordering codes. Bell Atlantic also conducts regular training courses for competing carriers in key areas. In addition, Bell Atlantic's "Systems Support Help Desk" provides a single point of contact for competing carrier reports of system outages and software defects and provides help to ensure that any problems are resolved as quickly as possible. We are further encouraged by Bell Atlantic's practice of evaluating the performance of its help desk call agents and, when necessary, replacing the tools available to them for analyzing information and resolving problems. Although KPMG reported confusion regarding contact lists and help desk numbers, we find that Bell Atlantic has since fixed this problem. Specifically, we note that in September 1999, Bell Atlantic posted on its web site a comprehensive and descriptive list of the different support features available to competing carriers, including the time of day these support functions are available. Accordingly, we find that Bell Atlantic provides efficient competitors a meaningful opportunity to compete by enabling them to understand how to implement and use all of the OSS functions available to them. Thus, we reject commenters' allegations that Bell Atlantic's technical assistance and help desk support is inadequate. e. Pre-Ordering 128. Based on the evidence in the record, we conclude that Bell Atlantic demonstrates that it provides nondiscriminatory access to OSS pre-ordering functions. Bell Atlantic offers requesting carriers an industry standard application-to-application pre-ordering interface that enables carriers to integrate pre-ordering and ordering functions. Through this and other pr