Please Note: CalConnect is not currently (2019) conducting Interoperabiity Testing as separate events; testing opportunities are available during our four-day conferences to those interested in doing testing with other systems and applications. If you are interested in opportunities to do testing with others please contact us at contact@calconnect.org.
This page is intended for individuals who have not previously participated in (or observed) an Interoperability Test Event (also called an IOP Test Event or an Interop) and Developers Forum. Its purpose is to allow you to prepare for the event and understand in advance what to expect and how to maximize the value of your participation.
-
What is the Interoperabiity Test Event?
The Interoperability Test Event (hereafter the "Test Event") is two things: (a) an opportunity for organizations and individuals who wish to test their calendaring and scheduling servers, clients, libraries and services against other products to test interoperability and find (and fix) bugs and errors, and (b) a Developers Discussion Forum where participants take place in very detailed and technical discussions that are not suitable for the Conference.
Testing participants may be testing new products or functions, capabilities and changes to existing code, or ongoing implementations of still-changing specifications, to ensure that the specifications actually work – and have an opportunity to change them in response to the testing before they become standards. The Developers Forum offers an opportunity for technical discussions and BOFs at the "implementation" level, as opposed to the standards and specifications level.
Please be aware that CalConnect does not currently do certification or conformance testing, as opposed to interoperability testing between "real" implementations, nor does it have a "reference implementation" of CalDAV or other calendaring standards.
The CalConnect Interoperability Test Event is the only open test event of its kind for calendaring and scheduling and related technologies. -
When and where are Test Events held?
CalConnect holds three Test Events per year as part of the three CalConnect Events. Each week-long CalConnect Event begins with a 2½ day CalConnect Interoperability Test Event, followed by a Technical Conference.
Although a number of factors determine the ultimate location of each CalConnect Event, each year we endeavor to hold at least one in North America and one in Europe; currently in "Winter" (January/February), "Spring" (May/June), and "Autumn" (September/October). Should CalConnect develop significant memership in other parts of the world, we could look to hold events in locations which facilitate participation by those members.
CalConnect Events are hosted by CalConnect member organizations, generally on their premises or in nearby public space. -
How is the Test Event structured and organized?
The Test Event usually occupies the first two and a half days of the CalConnect event week: all day Monday, all day Tuesday, and Wednesday until noon. It is broken into several segments, generally 1.5 or 2 hours long. Prior to the event, the Interop Manager will have recorded who (people and organizations) are going to participate, what specific things they want to test (e.g. Implicit Scheduling, VPOLL, basic CalDAV operations) and perhaps implementations they would particularly like to test against. This information is recorded on the TC-IOPTEST-TESTING-NOTES Etherpad. Information about accessing remote servers and testing considerations unique to the environment of the host are collected on the TC-IOPTEST-CONFIGS Etherpad, and are discussed on the pre-event TC-IOPTEST calls, to which all interested parties are invited.
Generally, at the start of the first day, the group will discuss who wants to test what and with whom, and the Interop Manager will assign initial testing pairs and sequences of testing pairs, and will reassign testers as needed and as people complete their tests. The joint discussion will be repeated periodically throughout the Test Event. The goal is to ensure that everyone gets to test what they want to test with those who are interested in testing with them. Information will also be available on how to use the CalDAV Tester Test Suite and the CalDAV Load Simulation Tool for CalDAV servers.
During testing, participants record their findings on the TC-IOPTEST-TESTING-NOTES Etherpad; following the test event the entire contents of the Etherpad is preserved as a record of the testing and findings for the event (this is available to CalConnect member representatives). The Etherpad is then re-initialized for the next Test Event.
The discussions forming the Developers Forum portion of the Test event are generally identified in advance and a discussion list of oipics is located in the Test Event section of the event web page. In addition, a reading list for first-time attendees is prepared covering the topics both of the Developers Forum and the Conference, and is circulated to all registered attendees of both prior to the event. -
Who participates in the Interoperability Test Events?
The Test Events are open to both members and non-members, although the non-member registration fee is somewhat different. All attendees must agree to abide by the testing agreement and relevant parts of the CalConnect member agreement whether or not they are representing members.
Members and non-members may also send an individual to a single Test Event as an Observer, at a reduced cost, to find out what is going on and to help prepare for involvement in future events. Observers at a test event cannot participate in the testing; however they are welcome to fully participate in the Developers Forum dicsussions. -
What tools are used?
The nature of the internet access at the event is up to our host as they provide these facilities, and we will be constrained by their internal security considerations and constraints. Generally we will be using logons on the host's internal or guest system; however if necessary due to constraints we may set up an internal wired LAN (perhaps with wifi access for mobile devices) which has no internet connection. Details of the test environment will be established in advance on the TC-IOPTEST calls if possible, or will be announced first thing Monday morning at the Test Event.
Testing resources include the CalDAV Tester Test Suite and the CalDAV Load Simulation Tool for CalDAV servers. In addition, specific test matrices or other material may be developed for a particular Test Event and distributed prior to or at the event.
During the conduct of the Test Event, participants make use of the CalConnect channels on Slack as a way to communicate with remote testers and each other, and the two Etherpads associated with the Test Events, TC-IOPTEST-TESTING-NOTES and TC-IOPTEST-CONFIGS. Participants are specifically asked to record testing experiences, bugs, etc. on the testing notes etherpad instead or in addition to ongoing threads on Slack, as that is potentially ephemeral and not well-suited to long notes.
A note for non-member participants: normally non-members cannot access the tools referenced above as they are in the members-only area of our website. However, when you register you will be offered a userid on our system to be used for participation in the Test Event. At the end of the event, your userids will be disabled unless your organization decides to join CalConnect. As noted above, your participation will be governed by the same rules as member representatives. -
What do I need to do to prepare?
Most importantly, you do not need to have completely working code! The point of interoperability testing is to help find bugs and problems, and as soon as you have something that can execute at all, exposing it to interoperability testing will speed up your testing cycle. Remember, everyone else is in the same situation you are (or has been and will be again).
In particular, you should- Review the web page for the Next Interoperability Test Event to see what the plans and intended testing scenarios are for the event (be aware that this page is updated frequently as the event approaches)
- Register for the Test Event as soon as you know you are planning to participate
- When you register, identify on the registration form
- Specific things you would like to test (see example above)
- Specific implementations you would like to test against, if any (if they aren't registered we'll try to get them involved)
- Request a CalConnect Userid if you don't already have one
- Register a Slack userid for the CalConnect Channels
- Subscribe to the TC-IOPTEST mailing list if you're not already on it (or ask the Executive Director to subscribe you)
- Review the TC-IOPTEST Etherpads mentioned previously
- If you will have remote servers to be tested from the interop, provide the needed information on the TC-IOPTEST-CONFIGS Etherpad
- Start attending the TC IOPTEST calls (generally scheduled on Friday at 1100 Eastern)
- If you have questions, ask them on the TC-IOPTEST calls or ask the Interop Manager
- If you find your testing interests change, let the Interop Manager know in advance
- Review the list of Discussion Topics on the test event section of the event web page
- When the reading list is circulated, review it to be sure you are up to date on topics of interest to you
-
What should I bring?
It sounds silly to say it, but remember to bring the laptop machine running your testing code, if you plan to test it locally (this is mostly directed at server implementations). Bring a long ethernet cable in case we set up an internal network at the Test Event (we should know by the time of the last TC-IOPTEST call prior to the event). -
How do I get started testing at the Test Event?
If this is your first time, the Interop Manager should have arranged for someone who is familiar with the event to act as a mentor for you if you wish. Your mentor can help you get started, introduce you to people, help solve roadblocks, and in general help ensure you get the most value out of the event that you and your organization can. If for any reason this isn't working, please talk to the Interop Manager or the Executive Director immediately and one of them will act as your mentor. In any case, be sure and speak up at the opening discussion Monday morning when testing pairs are chosen.
If you are feeling lost or out of the loop at the Test Event, this can easily happen in a group where the majority of them have been to many Test Events, are testing things they have worked on together, and seem to be speaking a private language sometime. One of your mentor's jobs is to help you become part of the group as quickly as possible.
Finally, don't be shy about interrupting people if you have questions or need more information. If the person just can't talk right then they will get back to you asap, but usually they will be able to take a few moments to help you. The Interop Manager and Executive Director are there to ensure everyone testing gets the best value they can from the event – and you'll find that everybody there feels the same way. -
What about after the Test Event?
- If your organization is not a CalConnect Member, your userid and access to the internal systems and Slack will be diasbled following the CalConnect Event.
- Fill out the post-event survey we send to all participants shortly afterwards. These surveys are instrumental in helping us understand how to improve future events and to meet the needs of our members.
- As noted above, after every CalConnect Event we publish a Conference Report and a est Event report. Please see CalConnect Event Reports. The contents of the TC-IOPTEST-TESTING-NOTES Etherpad will be posted in the OWL Document Repository under CalConnect/Conferences (available to member representatives).
- If your organization is not yet a member, join CalConnect! For information about joining, please see Membership Information and in particular Membership Categories and Fees. If you need more information or help convincing your organization, the Executive Director is eager to help. If you have decided not to join at this time, please share your reasons with the Executive Director. Feedback from those deciding not to join is important input to CalConnect future directions and policies.
-
Where can I find more information about the next Test Event?
You can always find the logistics page and other information about the next CalConnect Event (Test Event and Technical Conference) via the EVENTS section of the website. Once planning information about the Test Event and Conference is available, it will be on that page in the appriopriate section, see the navigation list at the top.
The registration forms are available from the EVENTS section and linked from the logistics and information page; or directly from CalConnect Event Registration and Payment Options. -
Where can I find more information about test events in the past?
After every CalConnect Event we publish a Conference Report and a Test Event report. Please see CalConnect Event Reports.
For the Members-only Test Event reports through February of 2011, see the Internal IOP Reports page in the Members Area of the website.
Following February 2011, we discontinued producing a specific internal Test Event report. Instead we are preserving the set of testing notes created by the participants during each Test Event. These may be found in the Document Repository. -
A few last items
The logistics for each upcoming CalConnect event is posted as early as possible on the event web page. This will include information on location, travel, conference hotel(s), venue, and so forth.
CalConnect provides morning and afternoon refreshment breaks and lunch each day. While breakfast is your own responsibility, we will have coffe and rolls available at 0800 each morning (the event usually starts at 0830). Coffee, tea and water are usually available throughout the day. In addition, there is a dinner Monday evening for all Test Event participants, hosted by CalConnect.
Dress code at CalConnect events is casual to business casual. -
Questions?
Please contact the Executive Director at:
Dave Thewlis
+1 707 840 9391 voice | +1 707 498 2238 mobile | +1 415 946 3454 fax