Germany Trip Report, June 1982

Title: Trip Report FPS DIS MTU

Cast of characters

FPS USA personnel:
Robert Wallace, Hero
Dan Gorton, Field Support Engineer
Dave Borer, S/W Applications Engineer

FPS Germany personnel:
Reinhardt Albrecht, Hardware support manager
Assorted others

MTU personnel:
Dr. Gerhard Sieger, Customer contact (in charge of the application using DIS system)
Lutz Koegler, Programmer responsible for coding the DIS system
Klaus Schedl, Dr. Siegers boss

Data Translation personnel:
Walt Bastow, Primary engineer in charge of DIS
Dan Sulivan, alt Bastows boss and familiar with some of the DIS hardware
Fred Molinari, President of Data Translation

Equipment:
AP120B and a DIS with:

• ADC
• DAC
• QBT
• ADC DPM
• DAC DPM

PDP 11/34 running with RSX 11M rev 3.2

Synopsis:

This is basically a story of an unsuspecting diagnostic software engineer who is sent to Munich, Germany to aid in the installation of a new and exciting FPS product (the DIS.) It tells of the adventures and mishaps that occur and hopefully develops a moral.

Act 1, The first week

Scene 1, Portland

Monday, April 26, 1982.

This scene opens with our Hero, myself lying in bed nursing a terrible cold. I receive a phone call from my boss, Rich Watson. He asks if I would like to go to Germany. I. ask when. He replies “Today?”

I reply “I think not.” He asks “Tomorrow?”

After much thought I decide to make the sacrifice. I crawl out of bed and begin to make arrangements for the trip.

I have two tapes made containing the current diagnostic software for the DIS. I get my plane tickets. The initial plans call for me to be gone for about one week. (The problem being reported by Dan Gorton is that DISVER, a system test program written by the applications group, is always displaying zeros instead of the converted analog to digital voltages being input.)

The question of money arises. At one time I had a company VISA card but had to return it in a company purge. Since I plan on being gone for a short time the figure of 500.00 dollars is suggested as adequate. Some premonition of impending disaster prompts me to double the figure to 1000.00 dollars.

Tuesday, April 27, 1982

I leave Portland International Airport bound for Munich and glory. The trip is uneventful except for a minor episode in New York. Being unfamiliar with the rigors of Germany Customs I had decided to play it by the book in bringing the the two magnetic tapes I had. In Portland I had the FPS traffic personnel give me paper work declaring that the tapes were for educational purposes only and were going to be brought back with me to the USA. Checking with Customs in New York I discover Federal Bureaucracy. It seems that the papers I have are not the correct forms. So I have to fill out others. I also am informed that if I don’t return the forms to Customs when I return that there is a fine involved.

Scene 2, Munich

Wednesday, April 28, 1982

I arrive in Munich at 9:00 am. Naturally German Customs is not the least bit interested in anything I am bringing into the country. I am met at the airport by Dan Gorton and Reinhardt Albrecht. I decide that there is little reason to go to the hotel right away, so we drive to the customers site. To get on site requires getting security to issue badges for us and a pass for the car. None of the security people speak English. (Before I am to go home the security people will be greeting me with smiles and “ahh… F.P.S.”)

I am taken back to the engineering lab where the hardware is. The working environment is far less than optimum. The room is narrow with a long workbench along one wall. The other side of the room has at the window end the AP120 and DIS in a rack. Next to the 120 is a rack with the 11/34 and two RLOL disk drives. The system does not have a tape drive. Next to the 11/34 is two Decwriters. Across from the AP, on the workbench, is a VT100 type terminal. The two outstanding aspects of the lab are the lack of room, especially near the DIS, and the lack of air conditioning.

I am introduced to Dr. Sieger and to Lutz Koegler. The morning is spent being brought up to date on the problem by Dan Gorton. My first impression is that the problem was likely a hardware related one. The software that they were running had been run in Portland. The symptoms of getting zeros displayed seemed to indicate that the conversion was either not getting started or not finishing and generating the interrupt that unloaded the data from the ADC DPM to the AP120 main data. My first guess was that the problem might be a bad PLA on the ADC.

Thursday, April 29, 1982

I spent the day probing the ADC, DPM and QBT cards trying to get an understanding of just what the hardware was suppose to be doing. One difficulty was the schematics that we had. They were not done in the standard FPS style, instead they were similar to DEC schematics. This made them very difficult to use and understand. There were no forward or backward references between signals and it was difficult to know exactly where a signal came from or where it went. During – this time the boards were being put on an extender and removed. (note: The customer supplied us with an extender card and MADE a extender cable for us! Nothing was shipped with the unit. No schematics were shipped with the unit. The schematics we were using were brought over to Munich by Dan Gorton. Not all of the schematics matched the rev levels of the boards. A big problem in using the ADC schematics was the the schematics referenced parts on the board by a letter ‘U’ number systen, ie U37. The silkscreen on the board however used a row column system, ie BS.)

Friday, April 30, 1982

During probing the boards and using GPDBUG I discovered that the IOCAL code was often halting at a location it should not be. Getting listings of the IOCAL code and the DIOR, and using GPDBUG I discover that the DIOR is hanging on a spin instruction. This instruction is executed during the DIOR’s instruction fetch of the next IOCAL instruction. A little more examination shows that the hang occurs on the instruction following the IOCAL VECTOR instruction. To test this hypothesis I replace the vector instruction with the equivalent code of four MOV instructions and run the program. It works!

Saturday, May 1, 1982

Visited the Deutsche Museum with Dan.

Sunday, May 2, 1982

Visited the concentration camp at Dachau. A very depressing place. Afterwards we visited the palace at Nymphenburg to cheer up.

Monday, May 3, 1982

DISVER still was running. I replaced the four MOV instructions with the VECTOR instruction and the program failed with the same symptoms as before. Putting the moves back in again resolved the problem. We demonstrated this the Dr. Sieger and Lutz Koegler.

The customer then showed us his application code. The code had been assembled in APAL with no errors but the assembled object code had jump instructions that had not assembled with the proper addresses. I looked into the code and the assembler and discovered that the problem was caused by using a FSUB instruction as a push for the adder. The problem was fixed by using FADDs for pushes. I called Portland and talked with Mark Haverkamp who told me that he also had run into this problem and that a TAR (technical action request) had been written on it.

The final problem the customer wanted us to look at was a test program Lutz Koegler had written to verify that the programmable clock card would work in the mode that they wanted to use it in. The program consisted of a FORTRAN mainline which started up a IOCAL routine that setup and started the programmable clock. The symptom was that the clock would start but then stop. The problem was that the FORTRAN mainline was not waiting for the GPIOP to stop running, instead the FORTRAN just started the GPIOP processing the IOCAL code and then did a exit to the operating system. I suspected that when the FORTRAN went back to the Operating system that a reset to the AP (perhaps unassigning the AP?) was occurring. The reset would stop the GPIOP. I placed a FORTRAN Pause statement before the program exit and the Programmable clock functioned properly.

Tuesday, May 4, 1982

I flew to Paris for some vacation time. I spent one week in Paris staying at the apartment of Michel and Viannette Rouillard. It was a very pleasant week that was spent exploring Paris on foot. Michel then invited me to spend another week on a 11 meter sail boat with himself and four of his friends. I hesitated, for I knew that duty called back in the States. This hesitation lasted for about twenty three nano-seconds, then I came to my senses and said yes. The next week was one of the most enjoyable weeks of my life. Fortunately during that time I did not know what lay ahead in Munich for me.

End of Act 1

Act 2, Munich again

Scene 1, Something hits the fan

Saturday, May 15, 1982

Arrived back in Paris. I found waiting for me a stack of messages inquiring as to my where abouts. The gist of the messages was that MTU was down and the customer was unhappy. I prepared to fly back to Munich.

Monday, May 17, 1982

I flew back to Munich in the evening. Hotel accommondations are arranged by the German FPS office with the Hilton hotel.

Tuesday, May 18, 1982

I first talk to Reinhardt Albrecht who is very concerned about the situation. He informs me that the customer is very angry and has been accusing FPS of manipulating the software so that the acceptance test would appear to run.

We go out to the site and talk with the customer. Lutz Koegler shows me a listing of DISVER (the ADC System test) and briefs me on errors in the code that he found. I examined the code and verified that there were programming errors in the code. The nature of the errors were such that they would not be seen while running the program. Lutz discovered them by trying to use DISVER as an example of programming the DIS. (He obviously had no experience with FPS software before or he probably would not have used that approach.)

There were two major errors in the code. One was that the code was suppose to preload the MUX for the first A/D transfer in chain mode. The comment indicated that this was happening but the actual code didn’t do this. The second error was in computing the starting buffer locations in the DPM. The way DISVER was calculating the second buffer caused it to overlap a portion of the first buffer. This was not apparent when DISVER was run because the data in the first and second buffer was always the same (being supplied by the test fixure). I felt that these errors were not serious and that the program still indicated the ADC functioned.

The problem was that when these errors where corrected and with the customers modifications to the code in place, the program didn’t give the correct outputs. With eight different channels being sampled and different voltages on each channel the output appeared such that the first channel was being sampled twice, or that the channels were shifted over one place. Spent more time checking the code line by line looking for more software mistakes. We didn’t find any more.

Spent the day probing the boards with the scope and logic analyzer. Discovered that two of the handshake signals from the ADC DPM going to the ADC weren’t terminated on the ADC. Being a versatile software engineer I whipped out a soldering iron from my back pocket (actually the customer Supplied me with a rather fancy digital controlled soldering iron) and added the two needed wires to the ADC board. This cleaned up the signals but had no effect on the original symptoms. The more I probed the more convinced I became that I was looking at a real hardware problem and that it Was probably beyond my scope (no pun intended). I called Wayne Matterson in Portland to report and suggested that Data Translation be called in.

Wednesday, May 19, 1982

Talked to Portland and was told that Data Translation was sending Dan Sulivan over to some other site and that he would leave Boston early to stop in at MTU to evaluate the situation. Then if needed, Walt Bastow was preparing to come over also. I spent the day probing the boards and writing test programs for scan, burst and PIO modes.

Thursday, May 20, 1982

Holiday in Germany. Dr. Sieger took me on a drive into the Alps.

This evening I discover that the Hilton doesn’t know that the German FPS office is picking up the bill. They seem to want some money. I try to explain that I don’t have any money. This seems to disturb them. I promise to have the bill taken care of tomorrow.

Friday, May 21, 1982

First thing I do is contact the German office about my problem with the hotel. They tell me that they had sent a telex to the Hilton that said to bill them, so they didn’t know what was wrong. But they assured me that they would take care of it.

Dan Sulivan arrived. He and I continued to look at the boards and isolated the problem to the ADC DPM. The problem had to do with. when the address counter was being clocked in the ADC scan mode. Dan first tried to adjust the timing by adding a delay line made with capacitors and resistors. We gave up on the delay line when we saw that we were not getting enough delay. That night looking at the problem on paper, Dan decided that the problem was not that the signal needed to be delayed but rather that the signal was being clocked on the wrong edge and to fix it merely required reversing the signal.

Saturday, May 22, 1982

We managed to work half a day at the site. This was difficult to do because of the problems with unions. Dan implemented his fix with a patch to the hardware and the test software appeared to run.

To celebrate Dan and I drove to Salzburg. On the way we pick up two women who were hitchhiking. They were sisters (Nuns). They were quite charming, the older one (about 63) spoke a little English. The other spoke only German. We drove them to the home where they live with four other sisters of their order. They lived near the Austrian boarder. We had coffee together and the Dan and I drove on to Salzburg.

Sunday, May 23, 1982

Walt Bastow arrived. Dan and I picked him up at the airport and we spent the evening examining different solutions to the ADC DPM problem that Dan had patched.

Monday, May 24, 1982

Walt removed the patches and put in a permanent fix. This fixed the ADC in scan mode. We hoped that the fix would also take care of any problems with the DAC. We began looking at the DAC running in scan mode. I worked on adding the DAC code to the test program. Things didn’t look good. The DAC didn’t work. The customers attitude began to change from hopeful to something less than that.

I have to check out of the Hilton and move into the Sheriton where Walt is staying. I contact the German FPS office to have them take care of the bill.

Tuesday, May 25, 1982

Walt and I continued to look at the problem. It appears to be related to the sequencer chip. Walt notes that the sequencer chip on the customers board is not a current rev level. He doesn’t think that that accounts for the problem but since we have a spare sequencer that is the new rev we decide to try it. The new sequencer didn’t work either but it changed some of the things we were looking at.

Wednesday, May 26, 1982

Still troubleshooting. Came in this morning and nothing worked the same as it did when we left last night: We spent the day trying to restore the system back to the same symptoms we were experiencing. The customers confidence is plummeting.

I think that this was the day that Fred Molinari arrived with a spare ADC module. The customers original ADC had broken down before I arrived and had been replaced with a spare from the German office. The original module had been a differential input module, while the spare was a single ended. Fred Molinari brought what was thought to be a differential input module with him, but when we got it it turned out to be another single ended one.

Thursday, May 27, 1982

We got the system to behave somewhat in the same manner as Tuesday. We spent most of the day working with the new rev sequencer. We finally decided that the new sequencer was bad (a failure in the component). .We went back to the old sequencer. Walt called Data Translation and had them send a replacement sequencer. We hooked all 32 lines of the logic analyzer to the old sequencer chip and related signals and begin trying to determine what the old sequencer is doing. (Later, Dr. Sieger confides to me that when he saw the analyzer hooked up like that to one I.C. he lost all confidence that the system would ever run.)

Friday, May 28, 1982

Walt is convinced that the problem lies inside the logic of the sequencer. We spend most of the day walking through the states of the sequencer by hand. We finally discover a bad logic state in the sequencer. Walt thinks he can fix it on site because it only requires blowing a fuse inside the Sequencer. We call Portland to find out the procedure and discover that it is too difficult to attempt. Walt called Data Translation and had a new sequencer chip made with the fix. Dr. Sieger invites Walt and I to go with him to his home in Koln (Cologne). He commutes on weekends between Koln and Munich, a distance of about 480 kilometers. The Siegers live in a small village about 15 kilometers from Koln near an large forested area. We spend a pleasant weekend with Dr. Sieger and his wife and children. Saturday afternoon we walk through the forest for about four kilometers to the cathedral at Attenberg.

Sunday, May 30, 1982

Margot (Dr. Siegers wife) is in Holland taking sailing lessons. Dr. Sieger, Walt and I visit the famous cathedral at Koln. We also visit the Roman museum next to the cathedral. Koln is one of the oldest cities in Germany, being built by the Romans. We then drove into Holland to meet Margot and have dinner. (It is humorous to see Dr. Sieger having the same problems with language with the Dutch that we have with Germans. )

Monday, May 31, 1982

Today is another holiday in Germany. We go for a walk through the forest to a small restaurant about ten kilometers from their house. To our surprise there is a Dixie Land band playing in the beer garden of the restaurant. Later that evening we start back for Munich.

Scene 2, Munich, another week

Tuesday, June 1, 1982

We receive the first ordered replacement sequencer chip. It does us no good though because we had discovered the bad programming of the chip. We spend the day working on PIO. We find a strange error in the IOCAL. The output of the customers application code has two different labels that have been assigned the same value. All other labels, before and after, are correct. We try the assembly on another system in a cooler room to verify that we aren’t seeing a memory or hardware problem in the PDP 11/34 and the same thing happens.

I notice that there are a large number of symbols and count them. The error occurs at the 100th symbol. [I get the source of IOCAL off the installation tape and discover that one of the symbol table arrays is not properly dimensioned. I fix it and recompile and task build IOCAL and the error is fixed.

Walt and I looked at PIO timing on the DAC and discover that you can’t use a CBX instruction to do a multi-word transfer to the DAC in PIO mode. The problems appears to be that although the DIOR checks the QBT for data being ready, the QBT can push data to the DAC faster than the DAC can handle it and the DIOR is not checking the DAC card for being ready.

I decide that that is something that can be examined upon my return to Portland (although at this point I was seriously doubting that I would ever see my homeland again.) I recode the IOCAL program so that instead of trying to do a multi-word CBX it does a Single word CBX inside a loop. This greatly reduces the efficiency of the transfer. (We measure a setup on each CBX of 50 micro seconds.) While Lutz answering questions.

Wednesday, June 2, 1982

The new sequencer arrives and it improves the symptoms with the scan mode transfer. We continue to look at the problem and discover that the DPM board will need another change so that it will handle the data properly for the DAC. Walt makes the hardware change and glory hallelujah everything seems to work. Walt documented all of the changes that he has made.

I think that this was the day that we finally received a differential input ADC from Data Translation. Walt installed it and verified that it was differential and that it worked.

Thursday, June 3, 1982

I take a close look at all of the programs we have worked on and discover that the program that I had thought was doing burst mode was really coded for scan mode. I change the code and rebuild it. Of course it didn’t work. Walt and I look into it and find that in burst mode if the buffer sizes are too small that the AP will not have time to empty the data from the DPM before the next buffer starts to overwrite the buffer. The AP can empty the buffer twice as fast as either the ADC or DAC can use them. But before the AP starts there is software overhead. We measure the overhead and try buffers of different sizes to find out what the minimum buffer size needs to be. We found that a buffer of 64 will work.

The Sheriton requests that I pay the bill. They don’t seem to know that the German FPS office is picking up the bill. They seem to want some money. I try to explain that I don’t have any money. This seems to disturb then. I promise to have the bill taken care of tomorrow.

Friday, June 4, 1982

First thing I do is contact the German office about my problem with the hotel. They tell me that they had sent a letter to the Sheriton that said to bill them, so they didn’t know what was wrong. But they assured me that they would take care of it.

I spent time verifying that the test programs still ran. I then worked with Lutz helping him with the code on their application. (The application was very interesting, the first program was modeling a jet engine, the second was also modeling a jet engine but with more detail.) I show Lutz some techniques to shorten his APAL code and write an algorithm for one aspect of the second model and then rough code the algorithm in APAL. Walt Spent some time talking with Dr. Sieger about how to use the DIS on future applications that will require high rates of throughput. The customer seems satisfied that the system is working. Walt decides to fly back to Boston tomorrow to be at his sons birthday party.

Saturday, June 5, 1982

I take Walt to the airport and say goodbye. I spend the day driving through the country side up into the Alps to Innsbruck. I am haunted all day with the feeling that I should continue through Austria and seek asylum in Switzerland. I return to Munich in the late evening.

Sunday, June 6, 1982

I worked on my notes for the trip report.

Scene 3, The Great Depression

Monday, June 7, 1982

I arrived on the site with the intent to verify that the system was still up and running and to say my goodbyes to the people there.

Upon entering the lab I noticed Dr. Sieger, Herr Schedl and Lutz clustered around the DIS looking at the scope. I was greeted by Dr. Sieger with “Have you ever checked the coding formats on the DAC?”. I sensed a trap and suspected that he already knew the answer. Sure enough, the coding formats (twos complement and binary offset) were not working properly. I silently cursed Walt Bastow for leaving me holding the bag.

I reopen the DIS and begin to look at the problem. The symptoms were that in twos complement format if a plus n volts was inputted the DAC would output a minus n volts. In offset binary format a zero voltage in gave plus and minus ten volts out. Plus nine mapped to plus one, plus eight mapped to plus two. Likewise on negative voltages.

The first thing I discover is that the documentation of the bit used to select the coding format is backwards. It indicated that if the bit was set the format was binary offset. Clear was twos complement. It should have been set for twos complement and clear for binary offset. When that was figured out it became obvious that something was apparently inverting the data. Since I couldn’t phone Data Translation until 14:00 hours I hooked up the scope and logic analyzer and began to look for something amiss. I probed around verifying that the circuit for handling twos complement and binary offset was correct but could not find anything wrong on the DAC. At 14:00 I called Walt at Data Translation. Walt returned my call and for the next six and a half hours I became a guided probe over the phone for Walt.

At 20:30 Data Translation said there was a problem. They said they needed to look at it and would get back to me the next day. Needless to say that the customer was a little upset. (Especially considering that the failure was discovered by Herr Schedl when Dr. Sieger went to demonstrate that the system was working.) I hypothesized that when we were looking at voltage levels on the scope, the scope must have accidentally been inverted and no one noticed. Lutz coded around the failure in his software.

Tuesday, June 8, 1982

I arrived on site and found Dr. Sieger talking with his boss, Klaus Schedl. After Herr Schedl left Dr. Sieger told me that his boss was very upset about everything that had happened and was planning to send a telex expressing his feelings to the German office. While I was waiting to hear from Data Translation I spent some time cleaning up the FORTRAN drivers for the test programs. I put the boards on extenders and tacked on some wires to several key signals so that we could take some measurements of the systems speed. The measurements came out very close to what Dr. Sieger had predicted based upon the system documentation.

At 14:30 I received a phone call from Walt Bastow. They discovered that they had used normal instead of inverting bus transceivers on the DAC. (For the path from the DPM to the DAC). Dr. Sieger checked and found that we could get the parts here and we decided to try to replace the parts tomorrow.

Just to make my day complete, Lutz showed me that the IOCAL was periodically returning an error code of “1”. This was interesting because that error code is never used anywhere in the system. He and I played with it and discovered that the error code being returned was the same as the number of words to read parameter in the call list. That is if the call to APGET (in the FORTRAN) requested only one word from the AP, then periodically we would not read the actual error code (which was always zero), but instead would get a one back from APGET. If a two or three word transfer was requested, then periodically the error code would be returned as a two or three.

I got the source of APGET and RUNDMA (which is called by APGET and examined them but could not see anything wrong. I replaced the call to APGET with a direct call to RUNDMA and found that the error still occurred. This indicated that the problem was in RUNDMA.

I told the customer that I would look into it in Portland. (I was beginning to feel that if I had to fix every problem the customer found with a FPS product, I would be there for the rest of my life. Of course that might not have been much longer if the customer found another problem).

Wednesday, June 9, 1982

Dr. Sieger and Herr Schedl decided not to risk replacing the bus transceivers. They felt that to replace two twenty pin I.C.’s on a six layer etch board was too prone to introducing other failures. They will continue to code around the problem until new boards arrive. I have Lutz make two tapes of the UFDs that have all of the test code we have been working with. The customer requests that I stay through Monday to insure that the system is still running next week.

Thursday, June 10, 1982

Another holiday. I ask Dr. Sieger about their vacation and holiday schedules. He tells me that most people get thirty days vacation and fifteen holidays a year. If a holiday falls on a Thursday then Friday is also a holiday. I tell him that we call that unemployment in the U.S.A. Dr. Sieger suggests that I check out of the Sheriton and move in with him. He has a nice flat near MIU in Dachau. I take him up on his offer and check out of the Sheriton. We drive back to Koln to spend the week end with his family.

Friday, June 11, 1982

Spend the day walking through the forests to several small restaurants. We return to their home and meet some of their friends for dinner.

Saturday, June 12, 1982

Another day spent in Koln. We meet the same friends as last night and go out to a small theater to see two french pantomime artists perform. Afterwards we have dinner at the friends house.

Sunday, June 13, 1982

Spend the day resting. We have a pleasant dinner and afterwards go to listen to a concert that the Siegers son is playing in. After that Dr. Sieger and I start on the drive back to Munich.

Monday, June 14, 1982

I check on the system and find that it is still running. The customer seems satisfied that it will do the application. (Please note that that sentence is different from “The customer is happy and the System is doing everything properly.”) I drive out to the German office and report in to Reinhardt Albrecht that the customer was releasing me from bondage and that I was planning to go home.

Tuesday, June 15, 1982

I have breakfast with Dr. Sieger and then say goodbye. My flight is scheduled out of Munich at 11:00. I go to the airport early to verify my tickets and discover that the rates have gone up starting today. To get home will cost another $157.67. A fitting ending. Fortunately I have enough cash on hand and pay the money. The next 20 hours are spent on planes and in airports by finally at 11:30 p.m. I arrived in Portland.

End of Act 2

Intermission

The following list of changes to the
boards at MTU are for sale in
the lobby. Also for sale, at a slightly
lower price is the list of ECO’s that
provide the permanent fixes.

Act 3, Reflections

This act consists of scenes done in surreal dreamlike Settings. Each scene contains a single theme.

Scene 1, The aftermath at the site

This scene has a stark Sterile setting, perhaps a empty stage with sharp lighting. A computerized voice off stage proceeds to itemize the outstanding problems that need to be resolved on the site.

Computer:

The following problems need to be resolved at MTU:

1) Inverting Bus transceivers need to be put on the DAC. (Walt Bastow may have returned to the site to do this)

2) GPDBUG hangs when trying to run the customers application code from the debugger.

3) APGET/RUNDMA intermittently returns incorrect value.

4) FSUB can’t be used as a push on the floating adder. This is a problem with the APAL assembler.

5) FMUL;LDDA;DB=value has a field conflict not detected by the APAL assembler. The FMUL is not done as a result.

6) The VECTOR instruction in DIOR causes a hang on the next instruction fetch. This is apparently a hardware timing problem in the GPIOP.

7) The DAC, ADC and DPM boards should be replaced with the new boards as soon as the new boards with the permanent fixes are available.

8) New F.P.S. software needs to bé installed. This software primarily consists of the full set of diagnostics for the DIS, a new DACVER and DISVER with the corrections, and new system tests for the untested modes of operation.

Scene 2, Reflections on the DIS

This scene has the stage dark with fog swirling around. A single light illuminates a podium where a man stands. The podium faces the audience.

Man:

I think that when the hardware is design verified, that the board swap philosophy will work. The design requirements of the diagnostics was not to verify and debug the hardware design, but to isolate hardware faults to a board level. At MTU we didn’t have a full set of diagnostics. Even if we had the situation would not have changed. The diagnostics would have failed even after swapping boards due to design flaws in hardware. The nature of the diagnostics (board level isolation) and the nature of the design flaws were such that the process of debugging the boards would have been the same.

The situation at MTU could have been avoided if Data Translations had design verified the hardware before shipping it to FPS. It also would have been avoided if FPS had had more time with the hardware before shipping the first units. The people working on the software (diagnostics and applications) were beginning to discover some of the design problems. Unfortunately this occured after the first units shipped and the problems were initially diagnosed as problems in understanding how to program the DIS because of the lack of clear documentation.

Three aspects of the DIS product.

a. Functionality.

After working with the DIS in the field with a customer, I feel the the hardware functionality is very good. Dr. -Sieger comment on several occasions (even during the- darkest hours) that he felt that IF we ever got the units working they would become one of our best products. Dr. Sieger, Walt Bastow and I talked about his applications for the units and it became apparent that the DIS is a very versatile data acquisition system.: The only drawback to this versatility is the complexity of the hardware.

b. Usability.

The hardware’s complexity and lack of good documentation make this product very difficult to use. It says something of the skill of Lutz Koegler in programming the DIS as far as he did. Especially when the DIS didn’t function properly. The major problem the user has with the DIS is that the actual hardware can not be invisible to the user. Ideally, the user should simply have to. say start an A/D conversion or start a D/A conversion and the hardware would do it. In the DIS the user must perform different steps depending on the direction of data transfers and the mode selection for the transfers. It requires knowledge of what the hardware is doing. This requires much better documentation and examples than currently exist.

Another aspect of usability is the lack of visibility the user has to the actual hardware. It became apparent in helping Dr. Sieger with his application code that being able to actual see on a scope or logic analyzer the hardware performing functions Was very important. This is especially true in real time data acquisition problems where the user needs to know exactly how much time is being used for the data acquisition and how much time is left for processing. Dr. Sieger mentioned that other systems he has seen have some front panel LEDs and/or scope connections to allow for this. We provided this capability by tacking on wires to some Signals and hanging the wires over the edge of the boards to where a logic analyzer could be attached. (When we were done the wires were removed.) If the boards were being re-etched I would Suggest that the possibility of bring some key signals to the edge of each board to some sort of connector and then having a front panel option with LEDs and Scope connections be examined.

c. Testability.

The testability of the DIS is very poor. There is little is the way of diagnostic visibility on the boards. Much of the hardware is mode dependent meaning that the basic hardware units will act differently depending upon the mode of operation the unit is configured in. I think that to insure fault detection and to provide better examples of programming the DIS, a library of system tests that approximate user applications needs to be developed. These tests would be dependent upon the particular configuration of DIS under test. For example: one test could be like the programs used at MTU which require a hardware configuration including a PCG, an ADC with DPM, and a DAC with DPM. This system test obviously could only be run on a system with the proper hardware configuration.

Scene 3, Customer attitudes

This scene has the main character slowly walking across the stage speaking to the audience. Behind him are large blowups of the marketing brochures for the DIS. His last lines are spoken just after he exits off stage.

Our primary contact with the customer was through Dr. Sieger, who was managing the development project that was using the AP system. His attitude towards FPS Germany was one of irritation. He said to me that he was told by the German office that there were many DIS units already in use, both in the U.S. and in Europe. This was obviously not the case. There seemed to be a poor relationship between the German offices sales group and the customer. Although I did talk to the salesman on the phone several times, I never saw him at the site. Dr. Sieger commented that the only time the salesman came to MTU was when it was time to sign off the equipment. (Dr. Sieger did say that the field support people he had experience with seemed to be very good.)

Dr. Sieger did help a great deal in the troubleshooting of the system. Not only did he supply us with test equipment and spare parts (I.C.s, cables, logic analyzers, etc.), but he also tried to keep pressure off of us. He expressed confidence in Walt and my capabilities and left us alone to do the work. (His confidence was shaken when Walt and I were trying to debug inside the sequencer chip.) I could see, however, his concern when things didn’t look good. He had committed his project to using the AP120 and DIS and if we could not get. it working he was going to be in trouble.

What was very nice was that he took a great deal of personal interest in Walt and I. He took us with him to his home in Koln. He took us to lunch and dinner many times. His bosses attitudes were a little difficult to determine because of the filtering that Dr. Sieger did. From some of the things said it was apparent that Herr Schedl did not think much of the situation and felt that it was very poor for FPS to be selling equipment in the state of the DIS.

Scene 4, Miscellaneous comments

1) There are mechanical difficulties with the board Clearance and the amount of force required to seat the boards. When removing or inserting boards you can hear the boards drag against each other (very disconcerting). The force required to seat boards sometimes cause the board to bend and warp.

2) All of the cables and connectors in the system need to be labeled and should have keys so that they can’t be inserted the wrong way. (Even Walt inserted a cable backwards once).

3) The front panel is designed so that when it is openned it hangs on the front at a ninety degree angle with all of the connectors on the bottom. At MTU is created irritation because we had to either fumble blindly trying to remove and install cables, or had to move the chairs out of the way and lay on our backs to install a cable. I would recommend that in addition to the way the front panel currently is designed to hang, that it be made so that it can hang vertically with the front connectors still facing front. (Hang it from the normal top connectors on the racks bottom pegs.)

4) More extensive documentation needs to be provided with programming examples for every type of mode. Also, timing diagrams need to be included in the documentation.

5) It would be valuable to have the capability of observing the actual data transfers on the front panel. This would require choosing signals on each board and bringing them to the edge of the board. From there they would need to be cabled to a front panel. I would Suggest that the front panel have LEDs and Scope connections so that if the system hangs it is apparent from the LEDs and the scope connections would give a customer the capability of measuring the efficiency of his system (including his S/W).

6) Diagnostic testing needs to be increased to include system level tests (That also could serve as examples) such as DISVER.

7) IOCAL at the site had several problems. The load module size was too small for the customer so I increased it. It had an array (SYMVL) dimensioned at 1/2 of what it should have been. Also, the comment on the structure of the symbol table was incorrect (N6+3 for the size should have been N7+3). These I fixed and will write a TAR for then.

8) The QBT card had all of its I.C.s reversed compared to the other cards in the DIS. Pin 1 on each I.C. on the QBT was at the upper left corner of the I.C. as you looked at the board On an extender, but all of the other cards the I.C.s were laid out so that Pin 1 was at the lower right corner. (I may have that backwards but that is not important.)

Act 4, Epilogue

Scene 1, Summation

This scene is the closing scene. The main character is sitting at the edge of the stage with his legs dangling over.

The original plan of F.P.S. for the prototype units was to create diagnostics for the QBT card and rely upon Data Translations tests for the other boards. The assumption was that these boards were standard products of Data Translation and should function properly. To give F.P.S. some confidence of the System, the applications group create a System test for the ADC and its DPM and for the DAC and its DPM. (Unfortunately the tests used the modes of operation that were functioning.).

During the development of the QBT diagnostics (before any prototypes were shipped) I went to Data Translation to bring up the QBT diagnostics. We were having problems with them that I attributed to our lack of understanding of the boards and the lack of good documentation. While at Data Translation Walt Bastow and I discovered some minor design flaws of the QBT dealing mostly with how -the board was being initialized. These were fixed and all of the QBT tests were checked out with some Simple fault insertion. Later, after the prototypes had been ship, during development of the remaining diagnostic tests I decided that I needed to return to Data Translation to finish bringing up the other tests.

We were experiencing the same types of difficulties with the other tests as were experienced with the QBT tests. Unfortunately I had planned to be at Data Translation only one week, After one and a half days I was sent to General Electric to aid them in installing their prototype DIS. This took the remainder of my planned time. (Walt and I again found some minor bugs during the day and a half. I also learned a great deal about programming the DIS and corrected many mistakes in the code.)

All of the hardware problems discovered at MTU should have been discovered at Data Translation. If not found at Data Translation, then they should have been found at F.P.S. The difficulty for F.P.S. was the lack of time prior to shipment, the lack of good documentation, and that the engineering prototype system was not even identical to the prototype systems being shipped.

Because F.P.S. assumed that the boards had been design verified, we assumed many of the difficulties that we were having were due to software errors which were due to lack of understanding about the system. Hindsight shows that a full set of design verification software should have been written prior to shipment and that time should have been given for that activity.

This should still be done and I feel that when F.P.S. as a running system in house again that fault insertion should be done to verify the diagnostics. Perhaps Walt Bastow should be brought to Portland to aid in the verification of the diagnostics. All of this points to the fact that there was little design verification done on the DIS system at Data Translation.

I think that another problem area during the project for F.P.S. was that there was very little Customer Service input to the project. I think that, especially on this type of project, that Customer Service should have acted as a quality watchdog. When the units shipped, no one in Customer Service knew anything about the hardware’ or the diagnostic software. They apparently were relying upon a board swap philosophy. While this philosophy is workable on a debugged design I still feel that if F.P.S. is selling the product there should be expertise on the product within Customer Service.

In the future, if another project like the DIS is begun, I would recommend that a much closer engineering relationship be established between whatever company is being contracted and F.P.S. to insure that the product and its documentation will meet the standards of F.P.S. I also feel that using another company for product development should not imply that the project time for F.P.S. is shortened by a great deal. In fact, I think that using another company may save some manpower, but may increase the projects schedule due to the problems of communication between the companies.

End of Act 4